### Example application.yml for Starting Canal Adapter

Source: https://github.com/alibaba/canal/blob/master/client-adapter/README.md

This is an example configuration for `application.yml` when starting the Canal adapter, including server port, logging level, and Canal server connection details.

```yaml
server:
  port: 8081
logging:
  level:
    com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  canalServerHost: 127.0.0.1:11111
  flatMessage: true
  canalInstances:
  - instance: example
    adapterGroups:
    - outAdapters:
      - name: logger

```

--------------------------------

### Start Canal Server

Source: https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart

Navigate to the Canal installation directory and execute the startup script to launch the Canal server.

```bash
cd /usr/local/canal/
sh bin/startup.sh
```

--------------------------------

### Canal Adapter Startup Configuration

Source: https://github.com/alibaba/canal/wiki/ClientAdapter

Example `application.yml` configuration for starting the Canal adapter, specifying server port, logging levels, and core adapter settings like Canal server host and mode. This configuration is used when launching the adapter service.

```yaml
server:
  port: 8081
logging:
  level:
    com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  canalServerHost: 127.0.0.1:11111
  batchSize: 500                            
  syncBatchSize: 1000                       
  retries: 0                               
  timeout:                                 
  mode: tcp 
  canalAdapters:                            
  - instance: example                       
    groups:                                 
    - groupId: g1                           
      outerAdapters:                        
      - name: logger                        

```

--------------------------------

### Start Kafka Server

Source: https://github.com/alibaba/canal/wiki/Kafka-QuickStart

Use the provided start script to launch the Kafka server in the background. Configuration is loaded from server.properties.

```bash
# bin/kafka-server-start.sh  -daemon  config/server.properties &
```

--------------------------------

### Start Canal (Linux)

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Start the Canal service on Linux using the startup.sh script.

```bash
sh startup.sh 
```

--------------------------------

### Copy Example Configuration

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Use this command to create new instance directories by copying the default example configuration. This is useful when setting up multiple Canal instances.

```bash
cp -R example example1/
cp -R example example2/
```

--------------------------------

### Start Canal Server with Local Configuration

Source: https://github.com/alibaba/canal/wiki/Canal-Admin-ServerGuide

Command to start the Canal server using the `canal_local.properties` file, enabling connection to Canal Admin.

```bash
sh bin/startup.sh local
```

--------------------------------

### Install canal-admin Helm Chart

Source: https://github.com/alibaba/canal/blob/master/charts/README.md

Installs the canal-admin chart using a specified values file.

```sh
helm install canal-admin -f ./admin-values.yaml ./canal-admin
```

--------------------------------

### Canal server startup log

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Example log output indicating a successful Canal instance startup.

```log
2013-03-19 18:18:20.590 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2013-03-19 18:18:20.596 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2013-03-19 18:18:20.831 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2013-03-19 18:18:20.845 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful...
```

--------------------------------

### Start Canal (Windows)

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Start the Canal service on Windows using the startup.bat script.

```batch
startup.bat
```

--------------------------------

### Start Zookeeper Service

Source: https://github.com/alibaba/canal/wiki/Zookeeper-QuickStart

Navigate to the Zookeeper bin directory and execute the zkServer.sh script to start the service.

```bash
# cd /usr/local/zookeeper/zookeeper-3.4.11/bin
# ./zkServer.sh  start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/zookeeper-3.4.11/bin/../conf/zoo.cfg
Starting zookeeper ...STARTED
```

--------------------------------

### Launch Canal Adapter Launcher

Source: https://github.com/alibaba/canal/wiki/Client-Adapter

Execute the canal-adapter-launcher.jar to start the client adapter. Ensure Canal server is running and configurations are set.

```bash
java -jar canal-adapter-launcher.jar
```

--------------------------------

### Compile ZK UI

Source: https://github.com/alibaba/canal/wiki/Zookeeper-QuickStart

Clone the ZK UI repository, install Maven, and compile the project using 'mvn clean install'. Ensure Maven is installed via yum.

```bash
#git clone https://github.com/DeemOpen/zkui.git
# wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
#cd zkui/
#yum install -y maven
#mvn clean install
```

--------------------------------

### Install canal-server Helm Chart

Source: https://github.com/alibaba/canal/blob/master/charts/README.md

Installs the canal-server chart using a specified values file. The server will automatically create and register itself with the admin cluster.

```sh
helm install canal-server -f ./server-values.yaml ./canal-server
```

--------------------------------

### Start Canal Adapter Script

Source: https://github.com/alibaba/canal/wiki/ClientAdapter

Shell script command to start the Canal adapter service. Ensure the `conf/application.yml` is correctly configured before execution.

```bash
bin/startup.sh
```

--------------------------------

### Start Canal Admin Service

Source: https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart

Execute the startup script to launch the Canal Admin service. Check the logs for confirmation of successful startup.

```bash
sh bin/startup.sh
```

--------------------------------

### Install Zookeeper Helm Chart

Source: https://github.com/alibaba/canal/blob/master/charts/README.md

Installs the Zookeeper Helm chart from the Bitnami registry. Refer to the Bitnami Zookeeper chart documentation for disk mounting options.

```sh
helm install canal-zookeeper oci://registry-1.docker.io/bitnamicharts/zookeeper
```

--------------------------------

### Start Canal instances on multiple machines

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Connect to each machine via SSH and execute the startup script to launch Canal instances.

```shell
-------
ssh 10.20.144.51
sh bin/startup.sh
--------
ssh 10.20.144.22
sh bin/startup.sh
```

--------------------------------

### MySQL Database Change Example

Source: https://github.com/alibaba/canal/wiki/ClientExample

This sequence of MySQL commands demonstrates creating a table and inserting a row, which would then be captured by the Canal client.

```sql
mysql> use test;
Database changed
mysql> CREATE TABLE `xdual` (
    ->   `ID` int(11) NOT NULL AUTO_INCREMENT,
    ->   `X` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
    ->   PRIMARY KEY (`ID`)
    -> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 ;
Query OK, 0 rows affected (0.06 sec)

mysql> insert into xdual(id,x) values(null,now());Query OK, 1 row affected (0.06 sec)
```

--------------------------------

### Install Canal Deployer

Source: https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart

Steps to download, copy, and extract the Canal deployer package. Ensure you use the latest version available.

```bash
mkdir -p /usr/local/canal
cp   canal.deployer-1.1.6.tar.gz   /usr/local/canal
tar -zxvf canal.deployer-1.1.6.tar.gz 
```

--------------------------------

### Run Canal Example Client with Maven

Source: https://github.com/alibaba/canal/wiki/ClientExample

This command executes the SimpleCanalClientTest class using Maven's exec plugin. Ensure Canal Server is running and accessible.

```bash
mvn exec:java -Dexec.mainClass="com.alibaba.otter.canal.example.SimpleCanalClientTest"
```

--------------------------------

### Start Canal-Adapter Launcher

Source: https://github.com/alibaba/canal/wiki/Sync-HBase

This command initiates the Canal-adapter service, which will then begin processing data synchronization tasks based on the configured `application.yml` and adapter-specific configuration files.

```bash
bin/startup.sh

```

--------------------------------

### Other SQL Examples for ES Sync

Source: https://github.com/alibaba/canal/blob/master/client-adapter/README.md

Examples include constructing geo-type fields from latitude and longitude, creating composite primary keys, and handling array fields.

```sql
select ... concat(IFNULL(a.latitude, 0), ',', IFNULL(a.longitude, 0)) AS location, ...
```

```sql
select concat(a.id,'_',b.type) as _id, ... from user a left join role b on b.id=a.role_id
```

```sql
select a.id as _id, a.name, a.role_id, c.labels, a.c_time from user a 
left join (select user_id, group_concat(label order by id desc separator ';') as labels from label
        group by user_id) c on c.user_id=a.id
```

```properties
objFields:
  labels: array:;
```

--------------------------------

### Canal Configuration Directory Structure

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Illustrates the typical directory structure for Canal deployment, including the main configuration file (canal.properties), instance configurations (example), and logging settings (logback.xml).

```bash
jianghang@jianghang-laptop:~/work/canal/deployer/target/canal$ ls -l conf/
总用量 8
-rwxrwxrwx 1 jianghang jianghang 1677 2013-03-19 15:03 canal.properties  ##系统配置
drwxrwxrwx 2 jianghang jianghang   88 2013-03-19 15:03 example  ## instance配置
-rwxrwxrwx 1 jianghang jianghang 1840 2013-03-19 15:03 logback.xml ## 日志文件
drwxr-xr-x 2 jianghang jianghang  168 2013-03-19 17:04 spring  ## spring instance目录
```

--------------------------------

### Composite Primary Key SQL Example

Source: https://github.com/alibaba/canal/wiki/Sync-ES

SQL example demonstrating the creation of a composite primary key by concatenating fields from joined tables.

```sql
select concat(a.id,'_',b.type) as _id, ... from user a left join role b on b.id=a.role_id
```

--------------------------------

### Simple Canal Client Example in Java

Source: https://github.com/alibaba/canal/wiki/ClientExample

A basic Java client implementation for Canal. It connects to a single instance, subscribes to all topics, and processes messages in batches. Includes error handling and acknowledgment.

```java
package com.alibaba.otter.canal.sample;

import java.net.InetSocketAddress;
import java.util.List;

import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.common.utils.AddressUtils;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.EventType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;

public class SimpleCanalClientExample {

    public static void main(String args[]) {
        // 创建链接
        CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress(AddressUtils.getHostIp(),
                                                                                            11111), "example", "", "");
        int batchSize = 1000;
        int emptyCount = 0;
        try {
            connector.connect();
            connector.subscribe(".*\\..*");
            connector.rollback();
            int totalEmptyCount = 120;
            while (emptyCount < totalEmptyCount) {
                Message message = connector.getWithoutAck(batchSize); // 获取指定数量的数据
                long batchId = message.getId();
                int size = message.getEntries().size();
                if (batchId == -1 || size == 0) {
                    emptyCount++;
                    System.out.println("empty count : " + emptyCount);
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                    }
                } else {
                    emptyCount = 0;
                    // System.out.printf("message[batchId=%s,size=%s] \n", batchId, size);
                    printEntry(message.getEntries());
                }

                connector.ack(batchId); // 提交确认
                // connector.rollback(batchId); // 处理失败, 回滚数据
            }

            System.out.println("empty too many times, exit");
        } finally {
            connector.disconnect();
        }
    }

    private static void printEntry(List<Entry> entrys) {
        for (Entry entry : entrys) {
            if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
                continue;
            }

            RowChange rowChage = null;
            try {
                rowChage = RowChange.parseFrom(entry.getStoreValue());
            } catch (Exception e) {
                throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(),
                                           e);
            }

            EventType eventType = rowChage.getEventType();
            System.out.println(String.format("================&gt; binlog[%s:%s] , name[%s,%s] , eventType : %s",
                                             entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
                                             entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
                                             eventType));

            for (RowData rowData : rowChage.getRowDatasList()) {
                if (eventType == EventType.DELETE) {

```

--------------------------------

### Start Canal with Debugging (Linux)

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Start Canal in debug mode on Linux, listening on a specified port. The 'suspend=y' option will block until a remote debugger attaches.

```bash
sh startup.sh debug 9099
```

--------------------------------

### Start Canal Server Docker Container

Source: https://github.com/alibaba/canal/wiki/Canal-Admin-Docker

Configure and launch Canal Server Docker containers, linking them to a Canal Admin instance. Use -e flags to set connection details for the admin manager, user, and password. The `canal.admin.register.cluster` option allows automatic registration into a specified cluster.

```bash
# Download the script
wget https://raw.githubusercontent.com/alibaba/canal/master/docker/run.sh 

# Start in standalone mode, connecting to Canal Admin
run.sh -e canal.admin.manager=127.0.0.1:8089 \
         -e canal.admin.port=11110 \
         -e canal.admin.user=admin \
         -e canal.admin.passwd=4ACFE3202A5FF5CF467898FC58AAB1D615029441
```

```bash
# Automatically join the 'test' cluster
run.sh -e canal.admin.manager=127.0.0.1:8089 \
         -e canal.admin.port=11110 \
         -e canal.admin.user=admin \
         -e canal.admin.passwd=4ACFE3202A5FF5CF467898FC58AAB1D615029441 
         -e canal.admin.register.cluster=test
```

--------------------------------

### Open Configuration File

Source: https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart

Use the vi command to open the instance.properties configuration file for editing.

```bash
vi conf/example/instance.properties
```

--------------------------------

### Geo Type SQL Example

Source: https://github.com/alibaba/canal/wiki/Sync-ES

Example SQL snippet for handling geo-type data by concatenating latitude and longitude.

```sql
select ... concat(IFNULL(a.latitude, 0), ',', IFNULL(a.longitude, 0)) AS location, ...
```

--------------------------------

### Elasticsearch Mapping for Multi-Table Example

Source: https://github.com/alibaba/canal/wiki/Sync-ES

This is the corresponding Elasticsearch mapping for the multi-table SQL example, defining the schema for the 'mytest_user' index.

```json
{
    "mytest_user": {
        "mappings": {
            "_doc": {
                "properties": {
                    "name": {
                        "type": "text"
                    },
                    "role_id": {
                        "type": "long"
                    },
                    "c_time": {
                        "type": "date"
                    },
                    "labels": {
                        "type": "text"
                    }
                }
            }
        }
    }
}
```

--------------------------------

### Initialize Canal Admin Database

Source: https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart

Connect to your MySQL instance and use the 'source' command to import the initialization SQL script. This creates the necessary database and tables for Canal Admin.

```bash
mysql -h127.0.0.1 -uroot -p

# 导入初始化SQL
> source conf/canal_manager.sql
```

--------------------------------

### Parent-Child Document Indexing - Elasticsearch Mapping Example

Source: https://github.com/alibaba/canal/wiki/Sync-ES

Example Elasticsearch mapping demonstrating the 'join' type for parent-child relationships between 'customer' and 'order'.

```json
{
  "mappings":{
    "_doc":{
      "properties":{
        "id": {
          "type": "long"
        },
        "name": {
          "type": "text"
        },
        "email": {
          "type": "text"
        },
        "order_id": {
          "type": "long"
        },
        "order_serial": {
          "type": "text"
        },
        "order_time": {
          "type": "date"
        },
        "customer_order":{
          "type":"join",
          "relations":{
            "customer":"order"
          }
        }
      }
    }
  }
}
```

--------------------------------

### Canal Table Filtering Regex Examples

Source: https://github.com/alibaba/canal/wiki/常见问题解答

Examples of Perl regular expressions for filtering tables in Canal. Ensure correct escaping of backslashes for complex patterns. These filters are only effective in row-based binlog mode.

```plain
.*   or  .*\\..*
```

```plain
canal\\..*
```

```plain
canal\\.canal.*
```

```plain
canal.test1
```

```plain
canal\\..*,mysql.test1,mysql.test2 (逗号分隔)
```

--------------------------------

### Stop Canal Server

Source: https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart

Navigate to the Canal installation directory and execute the stop script to shut down the Canal server.

```bash
cd /usr/local/canal/
sh bin/stop.sh
```

--------------------------------

### Launcher Directory Structure

Source: https://github.com/alibaba/canal/wiki/Client-Adapter

This structure outlines the expected files and directories for the canal-adapter-launcher.jar, including adapter dependencies and configuration files.

```text
canal-adapter-launcher.jar
- lib
    client-adapter.logger-1.1.1-jar-with-dependencies.jar
    client-adapter.hbase-1.1.1-jar-with-dependencies.jar
- config
    application.yml
    - hbase
        mytest_person2.yml
```

--------------------------------

### MariaDB 5.5 Binlog Event Definitions

Source: https://github.com/alibaba/canal/wiki/BinlogChange(MariaDB)

Defines the start of MariaDB-specific binlog events, with ANNOTATE_ROWS_EVENT being a key addition for RBR mode.

```java
/* New MySQL/Sun events are to be added right above this comment */
  MYSQL_EVENTS_END,

  MARIA_EVENTS_BEGIN= 160,
  /* New Maria event numbers start from here */
  ANNOTATE_ROWS_EVENT= 160,

  /* Add new MariaDB events here - right above this comment!  */

  ENUM_END_EVENT /* end marker */
```

--------------------------------

### HBase Adapter Configuration in application.yml

Source: https://github.com/alibaba/canal/blob/master/client-adapter/README.md

This configuration snippet shows how to set up the application.yml for the HBase adapter, including data source and Canal instance configurations.

```yaml
server:
  port: 8081
logging:
  level:
    com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  canalServerHost: 127.0.0.1:11111
  flatMessage: true
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true
      username: root
      password: 121212
  canalInstances:
  - instance: example
    adapterGroups:
    - outAdapters:
      - name: hbase
        properties:
          hbase.zookeeper.quorum: slave1
          hbase.zookeeper.property.clientPort: 2181
          zookeeper.znode.parent: /hbase

```

--------------------------------

### HBase Adapter Configuration in application.yml

Source: https://github.com/alibaba/canal/wiki/Client-Adapter

This configuration snippet shows how to set up the application.yml for the HBase adapter, including server port, logging levels, Jackson settings, Canal connection details, and specific HBase properties.

```yaml
server:
  port: 8081
logging:
  level:
    com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  canalServerHost: 127.0.0.1:11111
  flatMessage: true
  canalInstances:
  - instance: example
    adapterGroups:
    - outAdapters:
      - name: hbase
        properties:
          hbase.zookeeper.quorum: slave1
          hbase.zookeeper.property.clientPort: 2181
          zookeeper.znode.parent: /hbase

adapter.conf:
  datasourceConfigs:
    defaultDS:
      url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true
      username: root
      password: 121212
  adapterConfigs:
  - hbase/mytest_person.yml 
```

--------------------------------

### Canal client running node in ZooKeeper

Source: https://github.com/alibaba/canal/wiki/AdminGuide

Example of a ZooKeeper entry showing an active Canal client's connection details.

```zookeeper
[zk: localhost:2181(CONNECTED) 17] get /otter/canal/destinations/example/1001/running
{"active":true,"address":"10.12.48.171:50544","clientId":1001}
```

--------------------------------

### Configure Zookeeper for Multi-Node Cluster

Source: https://github.com/alibaba/canal/wiki/Zookeeper-QuickStart

For a multi-node setup, add server configurations to zoo.cfg, specifying the IP addresses and ports for each node.

```properties
server.1=192.168.1.110:2888:3888
server.2=192.168.1.111:2888:3888
server.3=192.168.1.112:2888:3888
```

--------------------------------

### Kafka SSL Configuration Parameters

Source: https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart

Set up Kafka SSL configuration in the canal.properties file, including enabling Kerberos and specifying Kerberos configuration file paths.

```properties
# canal.properties配置文件

kafka.kerberos.enable = true
kafka.kerberos.krb5.file = "../conf/kerberos/krb5.conf"
kafka.kerberos.jaas.file = "../conf/kerberos/jaas.conf"
```

--------------------------------

### Canal Adapter Directory Structure

Source: https://github.com/alibaba/canal/wiki/ClientAdapter

Illustrates the expected directory structure for a Canal adapter deployment, including bin, lib, plugin, conf, and logs directories. This structure is typically packaged into a tar.gz archive.

```text
- bin
    restart.sh
    startup.bat
    startup.sh
    stop.sh
- lib
   ...
- plugin 
    client-adapter.logger-1.1.1-jar-with-dependencies.jar
    client-adapter.hbase-1.1.1-jar-with-dependencies.jar
    ...
- conf
    application.yml
    - hbase
        mytest_person2.yml
- logs

```

--------------------------------

### Canal Adapter Launcher Directory Structure

Source: https://github.com/alibaba/canal/blob/master/client-adapter/README.md

This is the typical directory structure for the Canal adapter launcher, which is packaged as a compressed archive.

```bash
- bin
    restart.sh
    startup.bat
    startup.sh
    stop.sh
- lib
    client-adapter.logger-1.1.1-jar-with-dependencies.jar
    client-adapter.hbase-1.1.1-jar-with-dependencies.jar
    ...
- conf
    application.yml
    - hbase
        mytest_person2.yml
- logs

```

--------------------------------

### Create HBase Table

Source: https://github.com/alibaba/canal/wiki/Sync-HBase

This command is used in the HBase shell to create the target HBase table with a specified column family, which is required before starting data synchronization.

```sql
create 'MYTEST.PERSON', {NAME=>'CF'}

```

--------------------------------

### Multi-Table Mapping (One-to-Many) SQL Example

Source: https://github.com/alibaba/canal/wiki/Sync-ES

This SQL query demonstrates a one-to-many mapping scenario involving user and label tables. The subquery is restricted to a single table.

```sql
select a.id as _id, a.name, a.role_id, c.labels, a.c_time from user a 
left join (select user_id, group_concat(label order by id desc separator ';') as labels from label
        group by user_id) c on c.user_id=a.id
```

--------------------------------

### Download Canal Admin

Source: https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart

Use wget to download the Canal Admin package from the release page. Replace 'version' with the desired Canal version.

```bash
wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.admin-1.1.4.tar.gz
```

--------------------------------

### Set Zookeeper Node ID for Cluster

Source: https://github.com/alibaba/canal/wiki/Zookeeper-QuickStart

Create a 'myid' file in the data directory for each Zookeeper node, containing its unique ID (1, 2, or 3 in this example).

```bash
#master
echo "1">/usr/local/zookeeper/data/myid

#slave1
echo "2">/usr/local/zookeeper/data/myid

#slave2
echo "3">/usr/local/zookeeper/data/myid
```

--------------------------------

### Canal Client Update Event Log

Source: https://github.com/alibaba/canal/wiki/ClientExample

Example log output showing an UPDATE event for a row in the database. It details the before and after states of the row, including specific columns.

```log
================&gt; binlog[mysql-bin.002579:508882822] , name[retl,xdual] , eventType : UPDATE , executeTime : 1368607728000 , delay : 4270ms
-------&gt; before
ID : 1    update=false
X : 2013-05-15 11:43:42    update=false
-------&gt; after
ID : 1    update=false
X : 2013-05-15 16:48:48    update=true
```

--------------------------------

### Canal 实例时序表结构配置示例

Source: https://github.com/alibaba/canal/wiki/TableMetaTSDB

展示了在`instance.properties`文件中启用和配置时序表结构管理功能的具体参数，包括存储目录、JDBC URL、用户名和密码。

```properties
# table meta tsdb info
canal.instance.tsdb.enable=true
canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
canal.instance.tsdb.dbUsername=canal
canal.instance.tsdb.dbPassword=canal
```

--------------------------------

### Extract Canal Admin Archive

Source: https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart

Create a directory and extract the downloaded Canal Admin tarball into it. The '$version' placeholder should be replaced with the actual version number.

```bash
mkdir /tmp/canal-admin
tar zxvf canal.admin-$version.tar.gz  -C /tmp/canal-admin
```

--------------------------------

### REST API: Count Related Data

Source: https://github.com/alibaba/canal/wiki/ClientAdapter

curl command to get the total count of data for a specific adapter configuration, typically used for verification or monitoring purposes.

```bash
curl http://127.0.0.1:8081/count/hbase/mytest_person2.yml
```

--------------------------------

### Configure Canal Server and Adapters

Source: https://github.com/alibaba/canal/wiki/Sync-ES

Set up the Canal server host, batch sizes, retries, and data source configurations. Define the Elasticsearch adapter with its key, name, hosts, and connection properties like mode and cluster name.

```yaml
canal.conf:
  canalServerHost: 127.0.0.1:11111
  batchSize: 500
  syncBatchSize: 1000
  retries: 0
  timeout: 
  mode: tcp 
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true
      username: root
      password: 121212
  canalAdapters:
  - instance: example 
    groups:
    - groupId: g1
      outerAdapters:
      - 
        key: exampleKey
        name: es6                           # or es7
        hosts: 127.0.0.1:9300               # es 集群地址, 逗号分隔
        properties:
          mode: transport # or rest         # 可指定transport模式或者rest模式
          # security.auth: test:123456      # only used for rest mode
          cluster.name: elasticsearch       # es cluster name
```

--------------------------------

### View Canal Admin Logs

Source: https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart

Monitor the admin.log file to check the startup process and identify any potential errors. Successful startup is indicated by messages about Tomcat initialization and port listening.

```bash
vi logs/admin.log
```

--------------------------------

### Single Table Mapping SQL with Functions/Operations Example

Source: https://github.com/alibaba/canal/wiki/Sync-ES

Shows how to use SQL functions and arithmetic operations in the mapping query. Aliases are required for fields resulting from functions or operations.

```sql
select a.id as _id, concat(a.name,'_test') as name, a.role_id+10000 as role_id, a.c_time from user a
```

--------------------------------

### Download and Extract Kafka

Source: https://github.com/alibaba/canal/wiki/Kafka-QuickStart

Use wget to download the Kafka tarball, create a directory, copy the file, and then extract it.

```bash
wget https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.11-1.1.1.tgz
mkdir  -p /usr/local/kafka
cp   kafka_2.11-1.1.1.tgz   /usr/local/kafka
tar -zxvf kafka_2.11-1.1.1.tgz
```

--------------------------------

### Custom Instance XML Configuration

Source: https://github.com/alibaba/canal/wiki/DevGuide

Defines a custom Canal instance using Spring beans. This example shows how to configure a PropertyPlaceholderConfigurer to load properties and a CanalInstanceWithSpring bean with its dependencies.

```xml
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
    <property name="ignoreResourceNotFound" value="true" />
    <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/>
    <property name="locationNames">
        <list>
	    <value>classpath:canal.properties</value>
	    <value>classpath:${canal.instance.destination:}/instance.properties</value> 
        </list>
     </property>
</bean>

<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
    <property name="destination" value="${canal.instance.destination}" />
    <property name="eventParser">
	<ref local="eventParser" />
    </property>
    <property name="eventSink">
	<ref local="eventSink" />
    </property>
    <property name="eventStore">
        <ref local="eventStore" />
    </property>
    <property name="metaManager">
	<ref local="metaManager" />
    </property>
    <property name="alarmHandler">
	<ref local="alarmHandler" />
    </property>
</bean>
......
```

--------------------------------

### Configure MySQL Binlog

Source: https://github.com/alibaba/canal/wiki/QuickStart

Enable Binlog writing in MySQL and set the binlog-format to ROW. Ensure server_id is unique and does not conflict with Canal's slaveId.

```ini
[mysqld]
log-bin=mysql-bin #开启 binlog
binlog-format=ROW #选择 ROW 模式
server_id=1 #配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复
```

--------------------------------

### REST API: Get Data Synchronization Switch Status

Source: https://github.com/alibaba/canal/wiki/ClientAdapter

curl command to check the current status (on/off) of data synchronization for a specified Canal instance or MQ topic.

```bash
curl http://127.0.0.1:8081/syncSwitch/example
```

--------------------------------

### Configure Canal Server and RDB Adapter in application.yml

Source: https://github.com/alibaba/canal/wiki/Sync-RDB

This configuration sets up the Canal server host, batch sizes, and defines the source data source. It also specifies the RDB adapter for Oracle, including its JDBC driver, URL, credentials, and parallel thread count.

```yaml
canal.conf:
  canalServerHost: 127.0.0.1:11111
  batchSize: 500
  syncBatchSize: 1000
  retries: 0
  timeout: 
  mode: tcp #kafka rocketMQ
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true
      username: root
      password: 121212
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
      - name: rdb                                               # 指定为rdb类型同步
        key: oracle1                                            # 指定adapter的唯一key, 与表映射配置中outerAdapterKey对应
        properties:
          jdbc.driverClassName: oracle.jdbc.OracleDriver        # jdbc驱动名, 部分jdbc的jar包需要自行放致lib目录下
          jdbc.url: jdbc:oracle:thin:@localhost:49161:XE        # jdbc url
          jdbc.username: mytest                                 # jdbc username
          jdbc.password: m121212                                # jdbc password
          threads: 5                                            # 并行执行的线程数, 默认为1
```

--------------------------------

### Configure Canal Cluster Information

Source: https://github.com/alibaba/canal/blob/master/charts/README.md

Sets up cluster information, including Zookeeper address and server mode. This configuration is applied after the canal-admin pod starts and is accessible via the admin interface.

```yaml
canal.user = admin
canal.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441

canal.zkServers = canal-zookeeper.default.svc.cluster.local:2181
# tcp,kafka, rocketMQ, rabbitMQ, pulsarMQ 看实际需求，此处改为了rabbitMQ
canal.serverMode = rabbitMQ

#canal.instance.tsdb.dir = ${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url = jdbc:mysql://you_db_host:3306/canal_tsdb?useUnicode=true&characterEncoding=UTF-8&useSSL=false
canal.instance.tsdb.dbUsername = ****
canal.instance.tsdb.dbPassword = ****

#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml

#canal.instance.global.spring.xml = classpath:spring/file-instance.xml
canal.instance.global.spring.xml = classpath:spring/default-instance.xml

# 如果用的是阿里云polardb需要配置
canal.aliyun.accessKey = ****
canal.aliyun.secretKey = ****
canal.aliyun.uid = 主账号id

# 我们是推送到rabbitmq所以需要配置
rabbitmq.host = ****:5672
Rabbitmq.virtual.host = ****
Rabbitmq.exchange = ****
# canal.aliyun.accessKey
Rabbitmq.username = ****
# canal.aliyun.secretKey
Rabbitmq.password = ****
Rabbitmq.deliveryMode = direct

```