Maxwell

https://github.com/zendesk/maxwell
Admin
Maxwell's daemon, a mysql-to-json kafka producer
Tokens:15,712
Snippets:99
Trust Score:8.1
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Download and Install Maxwell CDC Tool

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Instructions for downloading and installing the Maxwell CDC tool using various package managers and direct download methods. This includes using `curl` for manual download and extraction, `docker` to pull the official image, and `homebrew` for macOS/Linux installations.

```curl
curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.43.2/maxwell-1.43.2.tar.gz \
       | tar zxvf -
cd maxwell-1.43.2
```

```docker
docker pull zendesk/maxwell
```

```shell
brew install maxwell
```

--------------------------------

### Start Basic Maxwell Table Bootstrap via Command Line

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/bootstrapping.md

Initiates a full data bootstrap for a specified table within a database using the `maxwell-bootstrap` command-line utility. This performs a `select *` and streams results.

```Shell
bin/maxwell-bootstrap --database fooDB --table barTable
```

--------------------------------

### Start Conditional Maxwell Table Bootstrap via Command Line

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/bootstrapping.md

Initiates a data bootstrap for a specified table, applying a SQL `WHERE` clause to filter the rows included in the bootstrap. Useful for replaying partial datasets based on specific criteria.

```Shell
bin/maxwell-bootstrap --database fooDB --table barTable --where "my_date >= '2017-01-07 00:00:00'"
```

--------------------------------

### Maxwell Bootstrapping Process and Replication Stream Output Example

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/bootstrapping.md

Illustrates the complete lifecycle of a Maxwell bootstrap, from table creation and initial data insertion to triggering the bootstrap and observing the resulting replication stream events. This includes `bootstrap-start`, `bootstrap-insert`, and `bootstrap-complete` event types alongside standard inserts.

```SQL
create table fooDB.barTable(txt varchar(255));
insert into fooDB.barTable (txt) values ("hello"), ("bootstrap!");
insert into maxwell.bootstrap (database_name, table_name) values ("fooDB", "barTable");
```

```JSON
{"database":"fooDB","table":"barTable","type":"insert","ts":1450557598,"xid":13,"data":{"txt":"hello"}}
{"database":"fooDB","table":"barTable","type":"insert","ts":1450557598,"xid":13,"data":{"txt":"bootstrap!"}}
{"database":"fooDB","table":"barTable","type":"bootstrap-start","ts":1450557744,"data":{}}
{"database":"fooDB","table":"barTable","type":"bootstrap-insert","ts":1450557744,"data":{"txt":"hello"}}
{"database":"fooDB","table":"barTable","type":"bootstrap-insert","ts":1450557744,"data":{"txt":"bootstrap!"}}
{"database":"fooDB","table":"barTable","type":"bootstrap-complete","ts":1450557744,"data":{}}
```

--------------------------------

### Install GPG using Homebrew

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Installs the GPG (GNU Privacy Guard) command-line tool on macOS using Homebrew, a popular package manager. GPG is essential for signing artifacts before publishing to Maven.

```shell
brew install gpg
```

--------------------------------

### Run Maxwell with Kafka Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Configure and run Maxwell to publish MySQL change data capture events to an Apache Kafka topic. This snippet includes examples for both direct command line execution and running within a Docker container, specifying Kafka bootstrap servers and topic.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \
   --producer=kafka --kafka.bootstrap.servers=localhost:9092 --kafka_topic=maxwell
```

```docker
docker run -it --rm zendesk/maxwell bin/maxwell --user=$MYSQL_USERNAME \
    --password=$MYSQL_PASSWORD --host=$MYSQL_HOST --producer=kafka \
    --kafka.bootstrap.servers=$KAFKA_HOST:$KAFKA_PORT --kafka_topic=maxwell
```

--------------------------------

### Configure Kafka Producer for High Throughput

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

This configuration example shows properties to set in `config.properties` for achieving high throughput performance with the Kafka producer. It sets `acks` to 1, enables `snappy` compression, and disables retries to prioritize speed over strict delivery guarantees.

```Properties
kafka.acks = 1
kafka.compression.type = snappy
kafka.retries=0
```

--------------------------------

### Configure Maven OSSRH Credentials and GPG Passphrase

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Provides an example `settings.xml` configuration for Maven, typically located in `~/.m2/settings.xml`. This file defines the OSSRH server credentials (username and password) and the GPG passphrase required for signing artifacts during the release process.

```xml
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">
    <localRepository/>
      <interactiveMode/>
      <offline/>
      <pluginGroups/>
    <servers>
    <server>
      <id>ossrh</id>
      <username>zendesk_maxwell</username>
      <password>ossrh_password_for_zendesk_maxwell</password>
    </server>
    </servers>
    <mirrors/>
    <proxies/>
    <profiles>
        <profile>
        <id>ossrh</id>
        <activation>
            <activeByDefault>true</activeByDefault>
        </activation>
        <properties>
            <gpg.executable>gpg</gpg.executable>
            <gpg.passphrase>gpg_key_passphrase</gpg.passphrase>
        </properties>
        </profile>
    </profiles>
    <activeProfiles/>
</settings>
```

--------------------------------

### Set Up AWS Credentials for Amazon SNS Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

This section explains the necessity of an IAM user with permissions to access the SNS topic for the Maxwell producer. It highlights the use of `DefaultAWSCredentialsProviderChain` for obtaining AWS credentials and provides references to AWS documentation for setup and region configuration.

```APIDOC
AWS Credentials for SNS:
  IAM User: Required with permissions to access SNS topic.
  Credential Provider: DefaultAWSCredentialsProviderChain
  AWS Docs Reference:
    - IAM user setup with Default Credential Provider Chain
    - Setting up a different region along with credentials
Producer SDK: AWS SNS SDK (com.amazonaws.services.sns.AmazonSNSClient)
```

--------------------------------

### Configure Kafka Producer for Reliable Delivery

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

This configuration example demonstrates properties for ensuring at-least-once delivery with the Kafka producer. It sets `acks` to `all` and configures retries to a larger number (e.g., 5) for increased reliability. Additionally, `min.insync.replicas` should be set on the Maxwell output topic for stronger guarantees.

```Properties
kafka.acks = all
kafka.retries = 5 # or some larger number
```

--------------------------------

### Configure Remote JMX Access for Maxwell

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/monitoring.md

Example shell command to set JAVA_OPTS environment variable for enabling remote JMX access to Maxwell. This configuration allows insecure connections without authentication, and requires setting SERVER_IP_ADDRESS.

```Shell
export JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9010 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=SERVER_IP_ADDRESS"
```

--------------------------------

### SQL Insert Statement for Maxwell Encryption Examples

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/encryption.md

A basic SQL INSERT statement used to demonstrate how data appears in Maxwell's output under different encryption settings. This statement populates the 'minimal' table with sample data.

```SQL
insert into minimal set account_id =1, text_field='hello'
```

--------------------------------

### Retrieve Maxwell Live Configuration State via HTTP GET

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

This API endpoint provides the current live configuration state of Maxwell. A GET request to this endpoint returns a JSON object containing the active filter configuration, allowing users to inspect the applied rules.

```APIDOC
GET: /config
{
	"filter": "exclude: noisy_db.*"
}
```

--------------------------------

### Start Maxwell HA Client Nodes with RAFT Member IDs

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/high_availability.md

Demonstrates how to start multiple Maxwell client nodes with High Availability enabled (`--ha`) and assign unique RAFT member IDs (`--raft_member_id`) to each. At least three nodes are required for RAFT-based leader election to function correctly, ensuring cluster resilience.

```bash
host1: $ bin/maxwell --ha --raft_member_id=A
host2: $ bin/maxwell --ha --raft_member_id=B
host3: $ bin/maxwell --ha --raft_member_id=C
```

--------------------------------

### MySQL Server Configuration for GTID-based Replication

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/deployment.md

Provides an example `my.cnf` configuration snippet to enable GTID mode for MySQL. This includes setting `server_id`, `log-bin`, `binlog_format`, `gtid-mode`, `log-slave-updates`, and `enforce-gtid-consistency` for robust GTID-based replication.

```MySQL Config
[mysqld]
server_id=1
log-bin=master
binlog_format=row
gtid-mode=ON
log-slave-updates=ON
enforce-gtid-consistency=true
```

--------------------------------

### MySQL Update and Maxwell Kafka JSON Output Example

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/index.md

This snippet demonstrates a MySQL update operation and the corresponding JSON output generated by Maxwell to Kafka. It illustrates how Maxwell captures database changes, including the database, table, type of operation, timestamp, new data values, and old data values.

```MySQL, JSON
  mysql> update `test`.`maxwell` set mycol = 55, daemon = 'Stanislaw Lem';
  maxwell -> kafka: 
  {
    "database": "test",
    "table": "maxwell",
    "type": "update",
    "ts": 1449786310,
    "data": { "id":1, "daemon": "Stanislaw Lem", "mycol": 55 },
    "old": { "mycol":, 23, "daemon": "what once was" }
  }
```

--------------------------------

### MySQL Update and Maxwell JSON Output Example

Source: https://github.com/zendesk/maxwell/blob/master/README.md

This snippet demonstrates a MySQL `UPDATE` statement and the corresponding JSON output generated by Maxwell. It illustrates how Maxwell captures data changes, including the database, table, type of operation, timestamp, new data, and old data values, for streaming to platforms like Kafka.

```MySQL
mysql> update `test`.`maxwell` set mycol = 55, daemon = 'Stanislaw Lem';
```

```JSON
{
  "database": "test",
  "table": "maxwell",
  "type": "update",
  "ts": 1449786310,
  "data": { "id":1, "daemon": "Stanislaw Lem", "mycol": 55 },
  "old": { "mycol": 23, "daemon": "what once was" }
}
```

--------------------------------

### Implement Custom Maxwell Filters with JavaScript `process_row` Function

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/filtering.md

Provides a JavaScript example for the `process_row` function, which allows for advanced filtering and data manipulation. It demonstrates updating a global `state` object, suppressing rows based on that state, and modifying row data (e.g., uppercasing a username) before Maxwell processes them.

```Javascript
function process_row(row, state) {
	// Updating the state object
	if ( row.database == "test" && row.table == "lock") {
		var haslock = row.data.get("haslock");
		if ( haslock == "false" ) {
			state.put("haslock", "false");
		} else if( haslock == "true" ) {
			state.put("haslock", "true");
		}
	}

	// Suppressing rows based on state
	if(state.get("haslock") == "true") {
		row.suppress();
	}

	// Filter and Change based on actual data
	if ( row.database == "test" && row.table == "bar" ) {
		var username = row.data.get("username");
		if ( username == "osheroff" )
			row.suppress();

		row.data.put("username", username.toUpperCase());
	}
}
```

--------------------------------

### Maxwell Bootstrap Utility Command-Line Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/bootstrapping.md

Reference for command-line options available with the `maxwell-bootstrap` utility, used to initiate data bootstrapping operations. Options control logging, MySQL connection details, target table, and client identification.

```APIDOC
maxwell-bootstrap Utility Options:
--log_level LOG_LEVEL: log level (DEBUG, INFO, WARN or ERROR)
--user USER: mysql username
--password PASSWORD: mysql password
--host HOST: mysql host
--port PORT: mysql port
--database DATABASE: mysql database containing the table to bootstrap
--table TABLE: mysql table to bootstrap
--where WHERE_CLAUSE: where clause to restrict the rows bootstrapped from the specified table
--client_id CLIENT_ID: specify which maxwell instance should perform the bootstrap operation
--comment COMMENT: arbitrary comment to be added to every bootstrap row record
```

--------------------------------

### Maxwell Configuration Priority and Methods

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

This section describes the different ways Maxwell can be configured, including command-line options, environment variables, and a properties file. It also specifies the priority order for these configuration sources, ensuring clarity on how settings are applied.

```APIDOC
Maxwell is configurable via the command-line, a properties file, or the environment.
The configuration priority is:
command line options > scoped env vars > properties file > default values

config.properties:
Maxwell can be configured via a java properties file, specified via `--config` or named "config.properties" in the current working directory. Any command line options (except `init_position`, `replay`, `kafka_version` and `daemon`) may be specified as "key=value" pairs.

via environment:
If `env_config_prefix` given via command line or in `config.properties`, Maxwell will configure itself with all environment variables that match the prefix. The environment variable names are case insensitive. For example, if maxwell is started with `--env_config_prefix=FOO_` and the environment contains `FOO_USER=auser`, this would be equivalent to passing `--user=auser`.
```

--------------------------------

### Create Kafka Topic with Specific Partitions

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

Provides a bash command to pre-create a Kafka topic with a specified number of partitions and a replication factor. This step is crucial as Maxwell discovers the topic's partition count upon boot, requiring the topic to exist beforehand.

```bash
bin/kafka-topics.sh --zookeeper ZK_HOST:2181 --create \
                    --topic maxwell --partitions 20 --replication-factor 2
```

--------------------------------

### List GPG Public and Private Keys

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Displays a list of all public and private GPG keys currently managed by the GPG keyring. This command is useful for verifying that your keys are correctly set up and for identifying your public key ID.

```shell
gpg --list-keys
```

--------------------------------

### Maxwell Core Configuration Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

This section outlines the primary configuration options available for Maxwell, detailing their arguments, descriptions, and default values. These options control various aspects of Maxwell's behavior, such as bootstrapping, binlog position, replay mode, memory usage, and HTTP configuration.

```APIDOC
option: bootstrapper
  argument: [async | sync | none]
  description: bootstrapper type. See bootstrapping docs.
  default: async
option: init_position
  argument: FILE:POSITION[:HEARTBEAT]
  description: ignore the information in maxwell.positions and start at the given binlog position. Not available in config.properties. [see note]
option: replay
  argument: BOOLEAN
  description: enable maxwell's read-only "replay" mode: don't store a binlog position or schema changes. Not available in config.properties.
option: buffer_memory_usage
  argument: FLOAT
  description: Determines how much memory the Maxwell event buffer will use from the jvm max memory. Size of the buffer is: buffer_memory_usage * -Xmx"
  default: 0.25
option: http_config
  argument: BOOLEAN
  description: enable http config endpoint for config updates without restart
  default: false
option: binlog_event_queue_size
  argument: INT
  description: Size of queue to buffer events parsed from binlog
  default: 5000
```

--------------------------------

### Run Maxwell with Google Cloud Pub/Sub Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Execute Maxwell to send MySQL CDC events to Google Cloud Pub/Sub. This requires specifying the Google Cloud project ID and the target Pub/Sub topic.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \
  --producer=pubsub --pubsub_project_id='$PUBSUB_PROJECT_ID' \
  --pubsub_topic='maxwell'
```

--------------------------------

### Execute Maxwell Release Script

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Navigates into the `./build` directory and runs the Maxwell release script using Bundler. This script automates the entire release process, including version bumping, tagging, and publishing to Maven. Replace `<the-new-version-number>` with the desired new version.

```shell
cd ./build
bundle exec ./release <the-new-version-number>
```

--------------------------------

### Configure GitHub Token in .netrc

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Illustrates the content for the `~/.netrc` file to store GitHub API credentials. This allows command-line tools to authenticate with `api.github.com` using a personal access token, which is required for certain release steps.

```shell
machine api.github.com
    login github-username
    password github-token
```

--------------------------------

### Configure Maxwell Column Filters to Exclude Rows by Value

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/filtering.md

Shows how to filter rows based on the stringified value of a specific column. This example rejects any row in `db.tbl` where the `col` column's value is 'reject'. Column filters are ignored if the specified column is not present in the row.

```Maxwell Configuration
--filter = 'exclude: db.tbl.col = reject'
```

--------------------------------

### Send Public GPG Key to Key Server

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Publishes a specified public GPG key to a public key server. This makes your key available for others to verify signatures or encrypt messages for you. Replace `<public-gpg-key>` with your actual GPG key ID.

```shell
gpg --send-key <public-gpg-key>
```

--------------------------------

### Run Maxwell with Redis Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Run Maxwell to publish MySQL CDC events to a Redis instance. This configuration requires specifying the Redis host.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \
    --producer=redis --redis_host=redis.hostname
```

--------------------------------

### Generate a Full GPG Key Pair

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Initiates the process to generate a new GPG key pair. This command prompts the user for key type, size, expiry, and user details. A passphrase can be set during this process, which will be needed later.

```shell
gpg --full-generate-key
```

--------------------------------

### Configure MySQL for Maxwell CDC

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Steps to configure your MySQL server to work with Maxwell. This involves setting `binlog_format=row` and `server_id` in `my.cnf` for proper binlog replication, and creating a dedicated MySQL user with necessary `GRANT` permissions for Maxwell to access binlogs and its own database.

```shell
# /etc/my.cnf

[mysqld]
# maxwell needs binlog_format=row
binlog_format=row
server_id=1 
log-bin=master
```

```mysql
CREATE USER 'maxwell'@'%' IDENTIFIED BY 'XXXXXX';
CREATE USER 'maxwell'@'localhost' IDENTIFIED BY 'XXXXXX';

GRANT ALL ON maxwell.* TO 'maxwell'@'%';
GRANT ALL ON maxwell.* TO 'maxwell'@'localhost';

GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'%';
GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'localhost';
```

--------------------------------

### Run Maxwell with Stdout Producer (Command Line)

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Execute Maxwell directly from the command line, configuring it to connect to MySQL and output change data capture events to standard output. This is useful for initial testing and debugging.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' --producer=stdout
```

--------------------------------

### Configure Google Cloud Pub/Sub Producer Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Outlines the extensive configuration parameters for the Google Cloud Pub/Sub producer. This includes topic and platform IDs, DDL event topics, batching thresholds (bytes, message count, delay), message ordering keys, and various retry and RPC timeout settings for robust message delivery. An emulator host can also be specified for local development.

```APIDOC
pubsub_topic:
  argument: STRING
  description: Google Cloud pub-sub topic
  default: N/A
pubsub_platform_id:
  argument: STRING
  description: Google Cloud platform id associated with topic
  default: N/A
ddl_pubsub_topic:
  argument: STRING
  description: Google Cloud pub-sub topic to send DDL events to
  default: N/A
pubsub_request_bytes_threshold:
  argument: LONG
  description: Set number of bytes until batch is send
  default: 1
pubsub_message_count_batch_size:
  argument: LONG
  description: Set number of messages until batch is send
  default: 1
pubsub_message_ordering_key:
  argument: STRING
  description: Google Cloud pub-sub ordering key template (also enables message ordering when set)
  default: N/A
pubsub_publish_delay_threshold:
  argument: LONG
  description: Set time passed in millis until batch is send
  default: 1
pubsub_retry_delay:
  argument: LONG
  description: Controls the delay in millis before sending the first retry message
  default: 100
pubsub_retry_delay_multiplier:
  argument: FLOAT
  description: Controls the increase in retry delay per retry
  default: 1.3
pubsub_max_retry_delay:
  argument: LONG
  description: Puts a limit on the value in seconds of the retry delay
  default: 60
pubsub_initial_rpc_timeout:
  argument: LONG
  description: Controls the timeout in seconds for the initial RPC
  default: 5
pubsub_rpc_timeout_multiplier:
  argument: FLOAT
  description: Controls the change in RPC timeout
  default: 1.0
pubsub_max_rpc_timeout:
  argument: LONG
  description: Puts a limit on the value in seconds of the RPC timeout
  default: 600
pubsub_total_timeout:
  argument: LONG
  description: Puts a limit on the value in seconds of the retry delay, so that the RetryDelayMultiplier can't increase the retry delay higher than this amount
  default: 600
pubsub_emulator:
  argument: STRING
  description: Google Cloud pub-sub emulator host to send events to
  default: N/A
```

--------------------------------

### Run Maxwell with Google Cloud BigQuery Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Run Maxwell to load MySQL change data capture events into Google Cloud BigQuery. This configuration requires specifying the BigQuery project ID, dataset, and table.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \
  --producer=bigquery --bigquery_project_id='$BIGQUERY_PROJECT_ID' \
  --bigquery_dataset='$BIGQUERY_DATASET' \
  --bigquery_table='$BIGQUERY_TABLE'
```

--------------------------------

### Configure Google Cloud Pub/Sub for Maxwell

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

This section details the requirements for publishing to Google Cloud Pub/Sub, including obtaining an IAM service account with the `roles/pubsub.publisher` role. It specifies the `pubsub_project_id` and `pubsub_topic` properties for `config.properties` and mentions the optional `ddl_pubsub_topic` for DDL updates. The producer uses the Google Cloud Java Library for Pub/Sub.

```APIDOC
Google Cloud Pub/Sub Configuration:
  IAM Service Account: Required with 'roles/pubsub.publisher' role.
  Configuration Properties:
    pubsub_project_id: Google Cloud Project ID
    pubsub_topic: Pub/Sub topic for output stream
    ddl_pubsub_topic: (Optional) Dedicated output topic for DDL updates
  Producer SDK: Google Cloud Java Library for Pub/Sub
```

--------------------------------

### Maxwell General Configuration Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Defines core operational settings for the Maxwell daemon, including configuration file location, logging verbosity, and environment variable prefixing.

```APIDOC
General Configuration Options:
  config:
    type: STRING
    description: location of config.properties file
    default: $PWD/config.properties
  log_level:
    type: [LOG_LEVEL](#loglevel)
    description: log level
    default: info
  daemon:
    type: BOOLEAN
    description: running maxwell as a daemon
    default: false
  env_config_prefix:
    type: STRING
    description: env vars matching prefix are treated as config values
    default: (none)
```

--------------------------------

### Configure MySQL Global Binlog Format and Image at Runtime

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/deployment.md

Demonstrates how to dynamically change MySQL's global `binlog_format` to `ROW` and `binlog_row_image` to `FULL` without a server restart. Note that `binlog_format` is session-based, requiring active connections to be shut down for full conversion.

```MySQL
mysql> set global binlog_format=ROW;
mysql> set global binlog_row_image=FULL;
```

--------------------------------

### Run Maxwell Release Script with Specific Step

Source: https://github.com/zendesk/maxwell/blob/master/build/README.md

Executes the Maxwell release script for a specific step, allowing manual control over the release process. This is useful for recovery from errors or for executing individual stages. Choose from `build`, `tag`, `release`, `changelog`, `docs`, or `nexus`.

```shell
bundle exec ./release <the-new-version-number> <build | tag | release | changelog | docs | nexus>
```

--------------------------------

### Run Maxwell with RabbitMQ Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Execute Maxwell to publish MySQL CDC events to a RabbitMQ message broker. This configuration requires specifying the RabbitMQ host.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \
    --producer=rabbitmq --rabbitmq_host='rabbitmq.hostname'
```

--------------------------------

### Maxwell Monitoring and Metrics Configuration Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Defines various configuration parameters for Maxwell's monitoring and metrics system. These options control aspects like metric prefixes, reporting mechanisms (SLF4J, JMX, HTTP, Datadog), enabling JVM metrics, and detailed settings for HTTP and Datadog integration, including server binding, diagnostic endpoints, intervals, API keys, and host/port configurations.

```APIDOC
option: metrics_prefix
  argument: STRING
  description: the prefix maxwell will apply to all metrics
  default: MaxwellMetrics

option: metrics_type
  argument: [slf4j | jmx | http | datadog]
  description: how maxwell metrics will be reported
  default: (None)

option: metrics_jvm
  argument: BOOLEAN
  description: enable jvm metrics: memory usage, GC stats, etc.
  default: false

option: metrics_slf4j_interval
  argument: SECONDS
  description: the frequency metrics are emitted to the log, in seconds, when slf4j reporting is configured
  default: 60

option: http_port
  argument: INT
  description: the port the server will bind to when http reporting is configured
  default: 8080

option: http_path_prefix
  argument: STRING
  description: http path prefix for the server
  default: /

option: http_bind_address
  argument: STRING
  description: the address the server will bind to when http reporting is configured
  default: all addresses

option: http_diagnostic
  argument: BOOLEAN
  description: enable http diagnostic endpoint
  default: false

option: http_diagnostic_timeout
  argument: MILLISECONDS
  description: the http diagnostic response timeout
  default: 10000

option: metrics_datadog_type
  argument: [udp | http]
  description: when metrics_type includes `datadog` this is the way metrics will be reported, can only be one of [udp | http]
  default: udp

option: metrics_datadog_tags
  argument: STRING
  description: datadog tags that should be supplied, e.g. tag1:value1,tag2:value2
  default: (None)

option: metrics_age_slo
  argument: INT
  description: Latency service level objective threshold in seconds (Optional). When set, a `message.publish.age.slo_violation` metric is emitted to Datadog if the latency exceeds the threshold
  default: (None)

option: metrics_datadog_interval
  argument: INT
  description: the frequency metrics are pushed to datadog, in seconds
  default: 60

option: metrics_datadog_apikey
  argument: STRING
  description: the datadog api key to use when metrics_datadog_type = `http`
  default: (None)

option: metrics_datadog_site
  argument: STRING
  description: the site to publish metrics to when metrics_datadog_type = `http`
  default: us

option: metrics_datadog_host
  argument: STRING
  description: the host to publish metrics to when metrics_datadog_type = `udp`
  default: localhost

option: metrics_datadog_port
  argument: INT
  description: the port to publish metrics to when metrics_datadog_type = `udp`
  default: 8125

option: custom_health.factory
  argument: CLASS_NAME
  description: fully qualified maxwell health check factory class, see example at https://github.com/zendesk/maxwell/blob/master/src/example/com/zendesk/maxwell/example/maxwellhealthcheckfactory/CustomMaxwellHealthCheckFactory.java
  default: (None)
```

--------------------------------

### Maxwell MySQL Connection and Replication Configuration

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Specifies parameters for connecting to MySQL, including host, user, password, port, JDBC options, SSL behavior, schema database, client ID, and replication settings. It also covers options for split server roles for replication and schema capture.

```APIDOC
MySQL Configuration Options:
  host:
    type: STRING
    description: mysql host
    default: localhost
  user:
    type: STRING
    description: mysql username
    default: (none)
  password:
    type: STRING
    description: mysql password
    default: (no password)
  port:
    type: INT
    description: mysql port
    default: 3306
  jdbc_options:
    type: STRING
    description: mysql jdbc connection options
    default: [DEFAULT_JDBC_OPTS](#jdbcopts)
  ssl:
    type: [SSL_OPT](#sslopt)
    description: SSL behavior for mysql cx
    default: DISABLED
  schema_database:
    type: STRING
    description: database to store schema and position in
    default: maxwell
  client_id:
    type: STRING
    description: unique text identifier for maxwell instance
    default: maxwell
  replica_server_id:
    type: LONG
    description: unique numeric identifier for this maxwell instance
    default: 6379 (see [notes](#multiple-maxwell-instances))
  master_recovery:
    type: BOOLEAN
    description: enable experimental master recovery code
    default: false
  gtid_mode:
    type: BOOLEAN
    description: enable GTID-based replication
    default: false
  recapture_schema:
    type: BOOLEAN
    description: recapture the latest schema. Not available in config.properties.
    default: false
  max_schemas:
    type: LONG
    description: how many schema deltas to keep before triggering compaction operation
    default: unlimited
  binlog_heartbeat:
    type: BOOLEAN
    description: enable binlog heartbeats to detect stale connections
    default: DISABLED

Replication Server Configuration:
  replication_host:
    type: STRING
    description: server to replicate from. See [split server roles](#split-server-roles)
    default: *schema-store host*
  replication_password:
    type: STRING
    description: password on replication server
    default: (none)
  replication_port:
    type: INT
    description: port on replication server
    default: 3306
  replication_user:
    type: STRING
    description: user on replication server
    default: (none)
  replication_ssl:
    type: [SSL_OPT](#sslopt)
    description: SSL behavior for replication cx cx
    default: DISABLED
  replication_jdbc_options:
    type: STRING
    description: mysql jdbc connection options for replication server
    default: [DEFAULT_JDBC_OPTS](#jdbcopts)

Schema Capture Server Configuration:
  schema_host:
    type: STRING
    description: server to capture schema from. See [split server roles](#split-server-roles)
    default: *schema-store host*
  schema_password:
    type: STRING
    description: password on schema-capture server
    default: (none)
  schema_port:
    type: INT
    description: port on schema-capture server
    default: 3306
  schema_user:
    type: STRING
    description: user on schema-capture server
    default: (none)
  schema_ssl:
    type: [SSL_OPT](#sslopt)
    description: SSL behavior for schema-capture server
    default: DISABLED
  schema_jdbc_options:
    type: STRING
    description: mysql jdbc connection options for schema server
    default: [DEFAULT_JDBC_OPTS](#jdbcopts)
```

--------------------------------

### Redis Producer Configuration Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

This section outlines the configuration parameters for the Redis producer, covering connection settings, authentication, database selection, and message types (Pub/Sub, Stream, or List). These options dictate how Maxwell interacts with Redis to publish data, allowing for flexible data distribution strategies.

```APIDOC
RedisProducerOptions:
  redis_host:
    type: STRING
    description: Host of Redis server
    default: localhost
  redis_port:
    type: INT
    description: Port of Redis server
    default: 6379
  redis_auth:
    type: STRING
    description: Authentication key for a password-protected Redis server
  redis_database:
    type: INT
    description: Database of Redis server
    default: 0
  redis_type:
    type: [ pubsub | xadd | lpush | rpush ]
    description: Selects either Redis Pub/Sub, Stream, or List.
    default: pubsub
  redis_key:
    type: STRING
    description: Redis channel/key for Pub/Sub, XADD or LPUSH/RPUSH
    default: maxwell
  redis_stream_json_key:
    type: STRING
    description: Redis XADD Stream Message Field Name
    default: message
  redis_sentinels:
    type: STRING
    description: Redis sentinels list in format host1:port1,host2:port2,host3:port3... Must be only used with redis_sentinel_master_name
  redis_sentinel_master_name:
    type: STRING
    description: Redis sentinel master name. Must be only used with redis_sentinels
```

--------------------------------

### Redis Producer Configuration Properties

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

This snippet details the configurable properties for the Redis producer, covering connection parameters, authentication, database selection, and the type of Redis data structure to use (pubsub, xadd, lpush, rpush). It also indicates default values and properties without defaults.

```Configuration
redis_type: pubsub, xadd, lpush, rpush (default: pubsub)
redis_key: maxwell (supports topic substitution)
redis_host: localhost
redis_port: 6379
redis_auth: null
redis_database: 0
redis_stream_json_key: message
redis_sentinels: (no default)
redis_sentinel_master_name: (no default)
```

--------------------------------

### Run Maxwell with Stdout Producer (Docker)

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Execute Maxwell within a Docker container, connecting to a MySQL instance and directing CDC events to standard output. This provides an isolated and portable way to run Maxwell.

```docker
docker run -it --rm zendesk/maxwell bin/maxwell --user=$MYSQL_USERNAME \
    --password=$MYSQL_PASSWORD --host=$MYSQL_HOST --producer=stdout
```

--------------------------------

### Maxwell Topic Substitution Variables Reference

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

Describes the available substitution variables for dynamically generating topics or equivalent identifiers in Maxwell producers like Kafka, Redis, and Nats. These variables are enclosed in `%{}` and allow for dynamic topic generation based on database, table, or operation type.

```APIDOC
Topic Substitution Variables:
  %{database}: Replaced with the database name.
  %{table}: Replaced with the table name.
  %{type}: Replaced with the operation type (insert/update/delete).

Available in Producers:
  - Kafka (for topics)
  - Redis (for channels)
  - Nats (for subject hierarchies)
```

--------------------------------

### Run Maxwell with NATS Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Run Maxwell to publish MySQL change data capture events to a NATS messaging system. This configuration specifies the NATS server URL for event delivery.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \
    --producer=nats --nats_url=='0.0.0.0:4222'
```

--------------------------------

### RabbitMQ Producer Configuration Properties

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

This snippet lists the configurable properties for the RabbitMQ producer, including connection details, exchange settings, and routing key templates. It specifies default values for most properties, with `rabbitmq_host` being the only required setting.

```Configuration
rabbitmq_host: (required)
rabbitmq_user: guest
rabbitmq_pass: guest
rabbitmq_virtual_host: /
rabbitmq_handshake_timeout: 10000
rabbitmq_exchange: maxwell
rabbitmq_exchange_type: fanout
rabbitmq_exchange_durable: false
rabbitmq_exchange_autodelete: false
rabbitmq_routing_key_template: %db%.%table%
rabbitmq_message_persistent: false
rabbitmq_declare_exchange: true
rabbitmq_use_ssl: false
```

--------------------------------

### Maxwell Kafka Producer Configuration Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Configuration options for Maxwell's Kafka producer, including Kafka broker addresses, topic names, dead-letter topic handling, Kafka version, partitioning hash functions, key formats, and DDL topic settings.

```APIDOC
- option: kafka.bootstrap.servers
  argument: STRING
  description: kafka brokers, given as `HOST:PORT[,HOST:PORT]`
  default: null
- option: kafka_topic
  argument: STRING
  description: kafka topic to write to.
  default: maxwell
- option: dead_letter_topic
  argument: STRING
  description: the topic to write a "skeleton row" (a row where `data` includes only primary key columns) when there's an error publishing a row. When `ignore_producer_error` is `false`, only RecordTooLargeException causes a fallback record to be published, since other errors cause termination. Currently only supported in Kafka publisher
  default: null
- option: kafka_version
  argument: [KAFKA_VERSION]
  description: run maxwell with specified kafka producer version. Not available in config.properties.
  default: 0.11.0.1
- option: kafka_partition_hash
  argument: default | murmur3
  description: hash function to use when choosing kafka partition
  default: default
- option: kafka_key_format
  argument: array | hash
  description: how maxwell outputs kafka keys, either a hash or an array of hashes
  default: hash
- option: ddl_kafka_topic
  argument: STRING
  description: if output_ddl is true, kafka topic to write DDL changes to
  default: *kafka_topic*
```

--------------------------------

### Common MySQL DML and DDL Operations

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/dataformat.md

Demonstrates basic SQL commands for creating, inserting, updating, deleting, altering, and dropping a table. These operations trigger corresponding data change events in Maxwell.

```SQL
create table test.e (
  id int(10) not null primary key auto_increment,
  m double,
  c timestamp(6),
  comment varchar(255) charset 'latin1'
);

insert into test.e set m = 4.2341, c = now(3), comment = 'I am a creature of light.';
update test.e set m = 5.444, c = now(3) where id = 1;
delete from test.e where id = 1;
alter table test.e add column torvalds bigint unsigned after m;
drop table test.e;
```

--------------------------------

### Maxwell SQS Producer Configuration Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Configuration options specific to Maxwell's SQS producer, for specifying the SQS Queue URI.

```APIDOC
- option: sqs_queue_uri
  argument: STRING
  description: SQS Queue URI
  default: null
```

--------------------------------

### Define Basic Jinja2 Template Layout

Source: https://github.com/zendesk/maxwell/blob/master/docs/maxwell_theme/main.html

This Jinja2 template snippet demonstrates core templating concepts: extending a `base.html` template, defining a `content` block, and including `toc.html` (table of contents) and `content.html` within that block. This pattern is commonly used for structuring web pages by reusing common layouts and modularizing content.

```Jinja2
{% extends "base.html" %} {%- block content %}

{% include "toc.html" %}

{% include "content.html" %}

{%- endblock %}
```

--------------------------------

### Run Maxwell with SNS Producer

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/quickstart.md

Execute Maxwell to publish MySQL CDC events to an Amazon SNS topic. This configuration requires specifying the SNS topic name and optional attributes to include in the message.

```shell
bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \
    --producer=sns --sns_topic=sns.topic --sns_attrs=database,table
```

--------------------------------

### Configure Maxwell Basic Filters with Include/Exclude Patterns

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/filtering.md

Demonstrates how to use the `--filter` flag to exclude all updates from `foodb` except for specific tables (`tbl`) and tables matching a regular expression (`/table_\d+/`). This allows fine-grained control over which updates are processed by Maxwell.

```Maxwell Configuration
--filter = 'exclude: foodb.*, include: foodb.tbl, include: foodb./table_\d+/'
```

--------------------------------

### Trigger Maxwell Bootstrap via SQL Insert into Bootstrap Table

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/bootstrapping.md

Demonstrates how to trigger a Maxwell bootstrap operation by inserting a row directly into the `maxwell.bootstrap` table. This method allows programmatic control over bootstrap tasks, including specifying client IDs and scheduling future runs.

```SQL
insert into maxwell.bootstrap (database_name, table_name) values ('fooDB', 'barTable');
```

```SQL
insert into maxwell.bootstrap (database_name, table_name, client_id) values ('fooDB', 'barTable', 'custom_maxwell_client_id');
```

```SQL
insert into maxwell.bootstrap (database_name, table_name, client_id, started_at) values ('fooDB', 'barTable', 'custom_maxwell_client_id', '2020-05-18 12:30:00');
```

--------------------------------

### Maxwell Custom Producer Configuration and Interface Implementation

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

Details the necessary steps to integrate a custom producer into Maxwell. This involves implementing the `ProducerFactory` interface and `AbstractProducer`, configuring the `custom_producer.factory` property with the fully qualified class name, and placing the producer's JAR in the `$MAXWELL_HOME/lib` directory. Custom properties for the producer can be accessed via `MaxwellConfig.customProducerProperties` using the `custom_producer.*` namespace.

```APIDOC
Maxwell Custom Producer Integration:
  Interfaces to Implement:
    - ProducerFactory: Responsible for creating AbstractProducer instances.
    - AbstractProducer: The custom producer logic.

  Configuration Properties:
    custom_producer.factory:
      Type: String (fully qualified class name of ProducerFactory)
      Description: Specifies the factory class for the custom producer.
    custom_producer.*:
      Type: Key-value pairs
      Description: Namespace for custom producer-specific configuration properties.

  Accessing Custom Properties:
    MaxwellConfig.customProducerProperties:
      Type: Properties object
      Description: Provides access to properties defined under custom_producer.*.
```

--------------------------------

### Configure Google Cloud BigQuery Producer Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Defines the essential configuration parameters for the Google Cloud BigQuery producer. This includes specifying the BigQuery project ID, dataset ID, and table ID where data will be ingested.

```APIDOC
bigquery_project_id:
  argument: STRING
  description: Google Cloud bigquery project id
  default: N/A
bigquery_dataset:
  argument: STRING
  description: Google Cloud bigquery dataset id
  default: N/A
bigquery_table:
  argument: STRING
  description: Google Cloud bigquery table id
  default: N/A
```

--------------------------------

### Configure Nats Output for Maxwell

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/producers.md

This section outlines the configurable properties for Nats integration with Maxwell. It specifies `nats_url` for the Nats server address and `nats_subject` for defining the Nats subject hierarchy, which supports topic substitution. Non-alphanumeric characters in substituted values are replaced by underscores.

```APIDOC
Nats Configuration Properties:
  nats_url: Nats server address (default: nats://localhost:4222)
  nats_subject: Nats subject hierarchy (default: %{database}.%{table})
    - Supports topic substitution.
    - Non-alphanumeric characters in substituted values are replaced by underscores.
```

--------------------------------

### Maxwell General Producer Configuration Options

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

Defines common configuration options applicable across various producer types in Maxwell, controlling aspects like producer selection, acknowledgment timeouts, partitioning strategies, and error handling.

```APIDOC
- option: producer
  argument: [PRODUCER_TYPE]
  description: type of producer to use
  default: stdout
- option: custom_producer.factory
  argument: CLASS_NAME
  description: fully qualified custom producer factory class, see [example](https://github.com/zendesk/maxwell/blob/master/src/example/com/zendesk/maxwell/example/producerfactory/CustomProducerFactory.java)
  default: null
- option: producer_ack_timeout
  argument: [PRODUCER_ACK_TIMEOUT]
  description: time in milliseconds before async producers consider a message lost
  default: null
- option: producer_partition_by
  argument: [PARTITION_BY]
  description: input to kafka/kinesis/sns/sqs partition function
  default: database
- option: producer_partition_columns
  argument: STRING
  description: if partitioning by 'column', a comma separated list of columns
  default: null
- option: producer_partition_by_fallback
  argument: [PARTITION_BY_FALLBACK]
  description: required when producer_partition_by=column. Used when the column is missing
  default: null
- option: ignore_producer_error
  argument: BOOLEAN
  description: When false, Maxwell will terminate on kafka/kinesis/pubsub publish errors (aside from RecordTooLargeException). When true, errors are only logged. See also dead_letter_topic
  default: true
```

--------------------------------

### Maxwell Configuration Value Types and Enums

Source: https://github.com/zendesk/maxwell/blob/master/docs/docs/config.md

This section defines specific value types and enumerated options used across Maxwell's configuration. It includes definitions for log levels, SSL options, producer types, JDBC options, partitioning strategies, and supported Kafka versions, providing the valid choices for each setting.

```APIDOC
LOG_LEVEL: [ debug | info | warn | error ]
SSL_OPTION: [ DISABLED | PREFERRED | REQUIRED | VERIFY_CA | VERIFY_IDENTITY ]
PRODUCER_TYPE: [ stdout | file | kafka | kinesis | pubsub | sqs | rabbitmq | redis ]
DEFAULT_JDBC_OPTS: zeroDateTimeBehavior=convertToNull&connectTimeout=5000
PARTITION_BY: [ database | table | primary_key | transaction_id | column | random ]
PARTITION_BY_FALLBACK: [ database | table | primary_key | transaction_id ]
KAFKA_VERSION: [ 0.8.2.2 | 0.9.0.1 | 0.10.0.1 | 0.10.2.1 | 0.11.0.1 ]
PRODUCER_ACK_TIMEOUT: In certain failure modes, async producers (kafka, kinesis, pubsub, sqs) may simply disappear a message, never notifying maxwell of success or failure. This timeout can be set as a heuristic; after this many milliseconds, maxwell will consider an outstanding message lost and fail it.
```
Tokens:
Raw