### Compile OpenLogReplicator via Docker Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc Commands to clone the repository and build the OpenLogReplicator Docker image. ```shell #> git clone https://github.com/bersler/OpenLogReplicator-docker #> bash build.sh ``` -------------------------------- ### Configure Database Logging Modes Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc SQL commands to verify and enable ARCHIVELOG mode and minimal supplemental logging, which are mandatory for redo-log-based replication. ```sql SELECT SUPPLEMENTAL_LOG_DATA_MIN, LOG_MODE FROM V$DATABASE; SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE ARCHIVELOG; ALTER DATABASE OPEN; ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; ALTER SYSTEM ARCHIVE LOG CURRENT; SELECT SUPPLEMENTAL_LOG_DATA_MIN, LOG_MODE FROM V$DATABASE; ``` -------------------------------- ### Running OpenLogReplicator (Bash) Source: https://context7.com/bersler/openlogreplicator/llms.txt These bash commands demonstrate how to run OpenLogReplicator using Docker or directly from a compiled binary. It covers starting the service with a configuration file, mounting volumes, exposing ports, and using command-line flags for version display and log level overrides. ```bash # Using Docker (recommended) docker run -d \ --name openlogreplicator \ -v /path/to/config:/opt/olr/config \ -v /mnt/oracle/redo:/mnt/oracle/redo:ro \ -v /var/lib/olr/checkpoint:/var/lib/olr/checkpoint \ -p 8080:8080 \ bersler/openlogreplicator:latest \ -f /opt/olr/config/OpenLogReplicator.json # Direct binary execution ./OpenLogReplicator -f /path/to/OpenLogReplicator.json # Display version and exit ./OpenLogReplicator --version # Run with custom log level (overrides config) ./OpenLogReplicator -f config.json # Disable root user check ./OpenLogReplicator -f config.json -r ``` -------------------------------- ### Manage Supplemental Logging for Tables and Database Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc SQL commands to enable primary key supplemental logging for specific tables or the entire database to ensure accurate change identification. ```sql -- Table level SELECT * FROM ALL_LOG_GROUPS WHERE LOG_GROUP_TYPE='PRIMARY KEY LOGGING' AND OWNER='' AND TABLE_NAME=''; ALTER TABLE .
ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS; -- Database level SELECT SUPPLEMENTAL_LOG_DATA_PK FROM V$DATABASE; ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS; ``` -------------------------------- ### Start OpenLogReplicator via CLI Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/reference-manual/reference-manual.adoc Demonstrates how to launch the OpenLogReplicator process using a custom configuration file and a specific process name for system monitoring. ```bash ./OpenLogReplicator -f ./configs/OpenLogReplicator.json -p OLReplicator ``` -------------------------------- ### Example Memory Configuration (JSON) Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/9.memory.adoc A sample JSON configuration demonstrating the 'memory' element with various parameters set. This example shows how to define memory limits, buffer sizes, and swap behavior for the OpenLogReplicator process. ```json { "memory": { "max-mb": 2048, "min-mb": 64, "read-buffer-min-mb": 8, "read-buffer-max-mb": 128, "swap-mb": 1536, "swap-path": "./tmp", "unswap-buffer-min-mb": 8, "write-buffer-min-mb": 16, "write-buffer-max-mb": 512 } } ``` -------------------------------- ### Enable Force Logging Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc SQL commands to enable FORCE LOGGING, ensuring all DML operations are captured in redo logs to prevent data loss. ```sql SELECT FORCE_LOGGING FROM V$DATABASE; ALTER DATABASE FORCE LOGGING; ALTER SYSTEM ARCHIVE LOG CURRENT; SELECT FORCE_LOGGING FROM V$DATABASE; ``` -------------------------------- ### OpenLogReplicator JSON Configuration Example Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/json.adoc A minimal example of the OpenLogReplicator JSON configuration file. This demonstrates the basic structure for defining source and target connections, including reader and writer configurations, and table filtering. ```json { "version": "1.9.0", "source": [ { "alias": "SOURCE", "name": "DBNAME", "reader": { "type": "online", "user": "user1", "password": "Password1", "server": "//host:1521/SERVICE" }, "format": { "type": "json" }, "filter": { "table": [ {"owner": "OWNER1", "table": "TABLENAME1", "key": "COL1,COL2,COL3"}, {"owner": "OWNER1", "table": "TABLENAME2"}, {"owner": "OWNER2", "table": "TAB.*"} ] } } ], "target": [ { "alias": "KAFKA", "source": "SOURCE", "writer": { "type": "kafka", "topic": "TOPIC", "properties": { "bootstrap.servers": "localhost:9092" } } } ] } ``` -------------------------------- ### Manage Supplemental Logging for Oracle Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc SQL commands to verify and enable supplemental logging for specific tables or the entire database. This is required for capturing UPDATE operations. ```sql SELECT * FROM ALL_LOG_GROUPS WHERE LOG_GROUP_TYPE='ALL COLUMN LOGGING' AND OWNER='' AND TABLE_NAME='
'; ALTER TABLE .
ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; SELECT * FROM ALL_LOG_GROUPS WHERE LOG_GROUP_TYPE='ALL COLUMN LOGGING' AND OWNER='' AND TABLE_NAME='
'; SELECT SUPPLEMENTAL_LOG_DATA_ALL FROM V$DATABASE; ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; SELECT SUPPLEMENTAL_LOG_DATA_ALL FROM V$DATABASE; ``` -------------------------------- ### Configure Table Matching and Tagging Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/6.table.adoc Provides a complete JSON structure example for configuring multiple table filters, including regex matching, surrogate keys, and output tagging. ```json { "table": [ { "owner": "user1", "table": "x_.*", "tag": "[all]" }, { "owner": "user1", "table": "y_.*", "key": "A,B,C", "tag": "A,B,C" }, { "owner": "user1", "table": "z_.*", "key": "[pk]", "tag": "[pk]", "condition": "([op] != 'd') && ([login username] != 'SYSTEM')" } ] } ``` -------------------------------- ### Example Target Configuration (JSON) Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/7.target.adoc This JSON snippet demonstrates a typical configuration for a 'target' element. It specifies the logical alias, the source it consumes data from, and the writer configuration for outputting replicated transactions. ```json { "alias": "audit_kafka", "source": "db_primary", "writer": { "type": "kafka", "topic": "db_audit", "properties": { "bootstrap.servers": "kafka1:9092,kafka2:9092" } }, "concurrency": 2 } ``` -------------------------------- ### Configure OpenLogReplicator JSON format settings Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/4.format.adoc An example JSON configuration object demonstrating the structure for defining format attributes, including timestamp precision, schema handling, and transaction ID formats. ```json { "format": { "type": "json", "attributes": 0, "char": 0, "column": 0, "flush-buffer": 1048576, "message": 1, "schema": 1, "timestamp": 8, "timestamp-type": 0, "user-type": 0, "unknown": 0, "xid": 0 } } ``` -------------------------------- ### Define Custom Supplemental Log Groups Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc SQL commands to create custom supplemental log groups for tables lacking a primary key. ```sql SELECT * FROM ALL_LOG_GROUPS WHERE LOG_GROUP_TYPE='USER LOG GROUP' AND OWNER='' AND TABLE_NAME='
'; ALTER TABLE .
ADD SUPPLEMENTAL LOG GROUP () ALWAYS; ``` -------------------------------- ### Example state configuration in JSON Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/11.state.adoc This JSON snippet demonstrates a typical configuration for the 'state' element in OpenLogReplicator. It specifies parameters for checkpoint storage, intervals, retention, and schema snapshotting. ```json { "state": { "type": "disk", "path": "./checkpoint", "interval-mb": 500, "interval-s": 600, "keep-checkpoints": 100, "schema-force-interval": 20 } } ``` -------------------------------- ### Configure File Writer with Rotation Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/8.writer.adoc Example configuration for the 'file' writer type, utilizing sequence IDs and file size limits for log rotation. This setup ensures files are managed based on size and formatted with timestamps. ```json { "type": "file", "output": "log_%i.txt", "max-file-size": 10485760, "new-line": 1, "write-buffer-flush-size": 4096 } ``` -------------------------------- ### Configure Source Element in JSON Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/1.source.adoc An example configuration snippet for the 'source' element. This structure defines the logical alias, connection name, and primary operational settings for the replication source. ```json { "source": { "alias": "db_primary", "name": "production_db", "arch": "online", "flags": 0, "redo-read-sleep-us": 50000, "refresh-interval-us": 10000000 } } ``` -------------------------------- ### Configure Prometheus Metrics in OpenLogReplicator Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/10.metrics.adoc A JSON configuration example demonstrating how to enable and bind the Prometheus metrics backend within the OpenLogReplicator configuration file. ```json { "metrics": { "type": "prometheus", "bind": "127.0.0.1:8080", "tag-names": "filter" } } ``` -------------------------------- ### Grant Database Privileges for Replication User Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc SQL script to grant the necessary SELECT and FLASHBACK privileges to a database user for accessing system metadata and redo logs. Must be executed by the SYS user. ```sql GRANT SELECT, FLASHBACK ON SYS.CCOL$ TO ; GRANT SELECT, FLASHBACK ON SYS.CDEF$ TO ; GRANT SELECT, FLASHBACK ON SYS.COL$ TO ; GRANT SELECT, FLASHBACK ON SYS.DEFERRED_STG$ TO ; GRANT SELECT, FLASHBACK ON SYS.ECOL$ TO ; GRANT SELECT, FLASHBACK ON SYS.LOB$ TO ; GRANT SELECT, FLASHBACK ON SYS.LOBCOMPPART$ TO ; GRANT SELECT, FLASHBACK ON SYS.LOBFRAG$ TO ; GRANT SELECT, FLASHBACK ON SYS.OBJ$ TO ; GRANT SELECT, FLASHBACK ON SYS.TAB$ TO ; GRANT SELECT, FLASHBACK ON SYS.TABCOMPART$ TO ; GRANT SELECT, FLASHBACK ON SYS.TABPART$ TO ; GRANT SELECT, FLASHBACK ON SYS.TABSUBPART$ TO ; GRANT SELECT, FLASHBACK ON SYS.TS$ TO ; GRANT SELECT, FLASHBACK ON SYS.USER$ TO ; GRANT SELECT ON SYS.V_$ARCHIVED_LOG TO ; GRANT SELECT ON SYS.V_$DATABASE TO ; GRANT SELECT ON SYS.V_$DATABASE_INCARNATION TO ; GRANT SELECT ON SYS.V_$LOG TO ; GRANT SELECT ON SYS.V_$LOGFILE TO ; GRANT SELECT ON SYS.V_$PDBS TO ; GRANT SELECT ON SYS.V_$PARAMETER TO ; GRANT SELECT ON SYS.V_$STANDBY_LOG TO ; GRANT SELECT ON SYS.V_$TRANSPORTABLE_PLATFORM TO ; ``` -------------------------------- ### Full Production Configuration (JSON) Source: https://context7.com/bersler/openlogreplicator/llms.txt This JSON configuration is designed for a complete production deployment of OpenLogReplicator. It includes settings for logging, memory management, metrics (Prometheus), state persistence, and defines both a source (Oracle) and a target (Kafka) with detailed parameters for each. ```json { "version": "1.9.0", "log-level": 3, "trace": 0, "dump-redo-log": 0, "dump-path": "/opt/olr/dump", "memory": { "min-mb": 128, "max-mb": 2048, "read-buffer-min-mb": 8, "read-buffer-max-mb": 128, "write-buffer-min-mb": 8, "write-buffer-max-mb": 256, "swap-mb": 1536, "swap-path": "/var/tmp/olr", "unswap-buffer-min-mb": 8 }, "metrics": { "type": "prometheus", "bind": "0.0.0.0:8080", "tag-names": "all" }, "state": { "type": "disk", "path": "/var/lib/olr/checkpoint", "interval-s": 600, "interval-mb": 500, "keep-checkpoints": 100, "schema-force-interval": 20 }, "source": [ { "alias": "PROD_SOURCE", "name": "PRODDB", "reader": { "type": "online", "user": "olr", "password": "SecurePassword123", "server": "//prod-db.company.com:1521/PRODDB", "path-mapping": ["/db/fra", "/mnt/oracle/fra"], "host-timezone": "+00:00", "log-timezone": "+00:00", "db-timezone": "+00:00" }, "format": { "type": "json", "timestamp": 8, "timestamp-type": 5, "scn": 1, "xid": 0, "rid": 1, "schema": 1, "column": 0, "message": 2, "db": 1, "flush-buffer": 1048576 }, "arch": "online", "flags": 0, "redo-read-sleep-us": 50000, "redo-verify-delay-us": 0, "refresh-interval-us": 10000000, "filter": { "separator": "|", "table": [ {"owner": "HR", "table": "EMPLOYEES", "key": "EMPLOYEE_ID"}, {"owner": "HR", "table": "DEPARTMENTS", "key": "DEPARTMENT_ID"}, {"owner": "SALES", "table": "ORDERS", "key": "ORDER_ID"}, {"owner": "SALES", "table": "ORDER_ITEMS", "key": "ORDER_ID,LINE_ITEM_ID"}, {"owner": "INVENTORY", "table": ".*"} ] } } ], "target": [ { "alias": "KAFKA_PROD", "source": "PROD_SOURCE", "writer": { "type": "kafka", "topic": "oracle-cdc-prod", "max-message-mb": 100, "poll-interval-us": 100000, "queue-size": 65536, "properties": { "bootstrap.servers": "kafka1:9092,kafka2:9092,kafka3:9092", "enable.idempotence": "true", "compression.codec": "snappy", "queue.buffering.max.messages": "200000", "retry.backoff.ms": "500", "message.send.max.retries": "5" } } } ] } ``` -------------------------------- ### ZeroMQ Output Configuration (JSON) Source: https://context7.com/bersler/openlogreplicator/llms.txt This JSON configuration sets up OpenLogReplicator to stream changes over ZeroMQ. It specifies a source database and configures the writer to use ZeroMQ for high-performance message delivery, including the ZeroMQ URI. ```json { "version": "1.9.0", "memory": { "min-mb": 64, "max-mb": 1024 }, "source": [ { "alias": "SOURCE", "name": "ORCL", "reader": { "type": "online", "user": "olr", "password": "SecurePassword123", "server": "//dbhost:1521/ORCL" }, "format": { "type": "protobuf" }, "filter": { "table": [ {"owner": "APP", "table": "EVENTS"} ] } } ], "target": [ { "alias": "ZEROMQ_TARGET", "source": "SOURCE", "writer": { "type": "zeromq", "uri": "tcp://0.0.0.0:5555" } } ] } ``` -------------------------------- ### CMake Project Setup and C++ Standards Source: https://github.com/bersler/openlogreplicator/blob/master/CMakeLists.txt Initializes the CMake build system, sets the project name, version, description, and homepage. It also enforces C++17 standard compliance and configures settings for macOS architectures and rpath. ```cmake cmake_minimum_required(VERSION 3.16) project(OpenLogReplicator VERSION 1.9.0 DESCRIPTION "OpenLogReplicator, Open Source Oracle database CDC" HOMEPAGE_URL "https://github.com/bersler/OpenLogReplicator" ) set(SCHEMA_VERSION "1.9.0") string(TIMESTAMP CMAKE_BUILD_TIMESTAMP "%Y-%m-%d %H:%M" UTC) set(CMAKE_CXX_STANDARD_REQUIRED TRUE) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_EXTENSIONS FALSE) if (OSX_ARCH) set(CMAKE_OSX_ARCHITECTURES ${OSX_ARCH}) endif () if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin") set(CMAKE_MACOSX_RPATH 1) set(MACOSX TRUE) endif() ``` -------------------------------- ### Configure Kafka Producer Properties Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/8.writer.adoc Example of defining additional Kafka producer properties within the 'properties' map. These settings allow fine-tuning of the librdkafka producer, such as bootstrap servers and compression codecs. ```json { "type": "kafka", "topic": "transactions", "properties": { "bootstrap.servers": "host1:9092,host2:9092", "compression.codec": "snappy", "message.send.max.retries": "3", "retry.backoff.ms": "500", "queue.buffering.max.ms": "1000", "enable.idempotence": "true" } } ``` -------------------------------- ### Create Dedicated Database User and Privileges Source: https://context7.com/bersler/openlogreplicator/llms.txt Creates a dedicated user account for OpenLogReplicator and grants the necessary SELECT and FLASHBACK privileges on system metadata tables. This ensures the tool has read-only access to the data dictionary required for parsing redo logs. ```sql -- Create user for OpenLogReplicator CREATE USER olr IDENTIFIED BY "SecurePassword123"; -- Grant required privileges on system tables GRANT SELECT, FLASHBACK ON SYS.CCOL$ TO olr; GRANT SELECT, FLASHBACK ON SYS.CDEF$ TO olr; GRANT SELECT, FLASHBACK ON SYS.COL$ TO olr; GRANT SELECT, FLASHBACK ON SYS.DEFERRED_STG$ TO olr; GRANT SELECT, FLASHBACK ON SYS.ECOL$ TO olr; GRANT SELECT, FLASHBACK ON SYS.LOB$ TO olr; GRANT SELECT, FLASHBACK ON SYS.LOBCOMPPART$ TO olr; GRANT SELECT, FLASHBACK ON SYS.LOBFRAG$ TO olr; GRANT SELECT, FLASHBACK ON SYS.OBJ$ TO olr; GRANT SELECT, FLASHBACK ON SYS.TAB$ TO olr; GRANT SELECT, FLASHBACK ON SYS.TABCOMPART$ TO olr; GRANT SELECT, FLASHBACK ON SYS.TABPART$ TO olr; GRANT SELECT, FLASHBACK ON SYS.TABSUBPART$ TO olr; GRANT SELECT, FLASHBACK ON SYS.TS$ TO olr; GRANT SELECT, FLASHBACK ON SYS.USER$ TO olr; GRANT SELECT ON SYS.V_$ARCHIVED_LOG TO olr; GRANT SELECT ON SYS.V_$DATABASE TO olr; GRANT SELECT ON SYS.V_$DATABASE_INCARNATION TO olr; GRANT SELECT ON SYS.V_$LOG TO olr; GRANT SELECT ON SYS.V_$LOGFILE TO olr; GRANT SELECT ON SYS.V_$PDBS TO olr; GRANT SELECT ON SYS.V_$PARAMETER TO olr; GRANT SELECT ON SYS.V_$STANDBY_LOG TO olr; GRANT SELECT ON SYS.V_$TRANSPORTABLE_PLATFORM TO olr; ``` -------------------------------- ### Configure Fast Recovery Area (FRA) Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc SQL commands to set the Fast Recovery Area size and destination, required for the OFFLINE reader type. ```sql ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE = 50G; ALTER SYSTEM SET DB_RECOVERY_FILE_DEST = '/opt/db/fra'; ``` -------------------------------- ### Configure Custom Key for OpenLogReplicator Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/installation/installation.adoc Defines a custom key for a specific table in the OpenLogReplicator configuration. This overrides the primary key and requires strict column existence validation. ```json {"owner": "", "table": "
", "key": ""} ``` -------------------------------- ### Example SQL Schema Definitions for XMLType Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/experimental-features/experimental-features.adoc These SQL statements demonstrate how to define tables with XMLType columns in Oracle Database. The first example (`USR1.ADAM1`) utilizes the database's native binary XMLType storage, while the second (`USR1.ADAM2`) explicitly stores the XMLType as a CLOB. ```sql CREATE TABLE USR1.ADAM1 ( A NUMBER(3), B XMLTYPE ); CREATE TABLE USR1.ADAM2 ( A NUMBER, B XMLTYPE ) XMLTYPE B STORE AS CLOB; ``` -------------------------------- ### Example debug configuration in JSON Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/3.debug.adoc This JSON snippet demonstrates a sample configuration for the 'debug' element in OpenLogReplicator. It includes parameters to control stopping the process based on log switches, checkpoints, and transactions, along with settings for owner and table names for debugging. ```json { "debug": { "stop-log-switches": 0, "stop-checkpoints": 0, "stop-transactions": 0, "owner": "TEST_SCHEMA", "table": "DEBUG_TABLE" } } ``` -------------------------------- ### Enable Prometheus Metrics (JSON Configuration) Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/metrics/metrics.adoc This snippet shows the JSON configuration required to enable Prometheus metrics in OpenLogReplicator. It specifies the metrics type as 'prometheus', the bind address and port for the metrics endpoint, and which tag names to include. Ensure Prometheus is configured to scrape this endpoint. ```json {"metrics": {"type": "prometheus", "bind": "0.0.0.0:8080", "tag-names": "all"}} ``` -------------------------------- ### Configure filter element in OpenLogReplicator JSON Source: https://github.com/bersler/openlogreplicator/blob/master/documentation/json/5.filter.adoc An example demonstrating the structure of the filter object, including table selection rules, a custom key separator, and transaction ID management for skipping or debugging. ```json { "filter": { "table": [ { "owner": "HR", "schema": "EMPLOYEES" }, { "owner": "ORDERS", "schema": "%", "key": "ORDER_ID", "tag": "orders" }, { "owner": "SYS", "schema": "%" } ], "separator": "|", "skip-xid": ["0x0002.012.00004162"], "dump-xid": [] } } ```