Apache Kafka Dockerfile

https://github.com/wurstmeister/kafka-docker
Admin
Dockerfile for Apache Kafka, providing a Docker image for running Kafka and Zookeeper with easy...
Tokens:13,742
Snippets:46
Trust Score:8.1
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Example Broker Configuration Output

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Illustrates the resulting Kafka broker configuration based on the provided listener configuration example.

```properties
advertised.listeners = OUTSIDE://ec2-xx-xx-xxx-xx.us-west-2.compute.amazonaws.com:9094,INSIDE://:9092
listeners = OUTSIDE://:9094,INSIDE://:9092
inter.broker.listener.name = INSIDE
```

--------------------------------

### Example Listener Configuration

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Provides a complete example of listener configuration for Kafka brokers, including hostname resolution, listener mapping, security protocols, and inter-broker communication settings.

```yaml
environment:
  HOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname
  KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
  KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
```

--------------------------------

### Kafka Client Bootstrap Metadata Example

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

An abbreviated example of metadata a Kafka client receives during bootstrap, showing broker information and topic partition assignments.

```json
{
  "brokers": [{"id":1001,"host":"one.prod.com","port":9092}],
  "topics": {
    "test": [
      {"partition":0,"leader":1001,"replicas":[1001,1002],"isrs":[1001,1002]},
      {"partition":1,"leader":1002,"replicas":[1002,1003],"isrs":[1002,1003]},
      {"partition":2,"leader":1003,"replicas":[1003,1001],"isrs":[1003,1002]}
    ]
  }
}
```

--------------------------------

### Single Broker Deployment

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Demonstrates how to deploy a single Kafka broker using a specific Docker Compose configuration file. This is useful for testing or specific single-node setups.

```yaml
docker-compose -f docker-compose-single-broker.yml up
```

--------------------------------

### Consume Kafka Topic with Kafkacat

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Example command to consume messages from a Kafka topic using kafkacat. It specifies broker addresses and the topic name.

```shell
kafkacat -b 192.168.1.2:32000,192.168.1.2:32001 -C -t test
```

--------------------------------

### Run All Kafka Docker Tests

Source: https://github.com/wurstmeister/kafka-docker/blob/master/test/Readme.md

This snippet shows how to set up the necessary Docker containers (Zookeeper, Kafka) and then execute all available tests using a shell script. It's the primary method for validating the entire Kafka Docker setup.

```shell
cd test
docker-compose up -d zookeeper kafka_1 kafka_2
./runAllTests.sh
```

--------------------------------

### Run Single Kafka Broker with Docker Compose

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Starts a single Kafka broker and a Zookeeper instance using docker-compose. This setup demonstrates basic Kafka connectivity within a Docker network, mapping container ports to the host.

```bash
# KAFKA_ADVERTISED_HOST_NAME: localhost
# ZOOKEEPER_CONNECT: zookeeper:2181
docker-compose -f docker-compose-single-broker.yml up -d
```

--------------------------------

### Configure Kafka JMX Options

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Example configuration for enabling and configuring JMX access for Kafka brokers. This involves setting JVM options to expose JMX ports and specify the RMI hostname.

```shell
KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099"
JMX_PORT: 1099
```

--------------------------------

### Docker Compose Cluster Management

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Provides essential Docker Compose commands for managing a Kafka cluster. This includes starting the cluster in detached mode, scaling the number of Kafka brokers, and stopping the cluster.

```yaml
docker-compose up -d
```

```yaml
docker-compose scale kafka=3
```

```yaml
docker-compose stop
```

--------------------------------

### Docker Compose Port Configuration for Swarm Mode

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Example of a Docker Compose port definition for deploying Kafka in Docker Swarm mode. Using 'host' mode for published ports ensures direct routing to brokers, avoiding ingress load balancing issues.

```yaml
ports:
  - target: 9094
    published: 9094
    protocol: tcp
    mode: host
```

--------------------------------

### Scale Kafka Brokers with Docker Compose

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Scales the Kafka deployment to multiple brokers and starts Zookeeper using docker-compose. This configuration highlights how to handle port mapping for multiple brokers by exposing ephemeral ports on the host.

```bash
# KAFKA_ADVERTISED_HOST_NAME: 192.168.1.2
# ZOOKEEPER_CONNECT: zookeeper:2181
docker-compose up -d zookeeper
docker-compose scale kafka=2
```

--------------------------------

### Consume Kafka Messages with KafkaCat

Source: https://github.com/wurstmeister/kafka-docker/wiki/SASL-config-example

This command uses Docker to run kafkacat in consumer mode, allowing you to read messages from a Kafka topic. It specifies the broker address, security protocol, SASL mechanism, username, and password for authentication. The output is formatted in JSON, and it starts consuming from the beginning of the topic.

```shell
docker run --rm -it solsson/kafkacat -C   -b localhost:9092  -X security.protocol=SASL_PLAINTEXT -X sasl.mechanism=SCRAM-SHA-256  -X sasl.username=user -X sasl.password=password   -t topic  -o beginning -J
```

--------------------------------

### Configure Kafka Topic Creation

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Defines environment variables for automatically creating Kafka topics upon broker startup. Specifies topic name, partition count, and replica count, with options for custom cleanup policies and separators.

```APIDOC
KAFKA_CREATE_TOPICS: "<topic_name>:<partitions>:<replicas>[:<options>]"
  - Creates Kafka topics automatically.
  - Example: "Topic1:1:3,Topic2:1:1:compact"
  - Options can include cleanup policies like 'compact'.

KAFKA_CREATE_TOPICS_SEPARATOR: "<separator>"
  - Overrides the default comma (`,`) separator for topic definitions.
  - Useful for multi-line YAML or complex topic configurations.
  - Example: "$$'\n'" to use newline as a separator, requiring ANSI-C quoting.
```

--------------------------------

### Publish Messages using Kafkacat

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Demonstrates how to publish messages to Kafka topics using the kafkacat command-line utility. It specifies the advertised host and ephemeral ports for multiple Kafka brokers.

```bash
kafkacat -b 192.168.1.2:32000,192.168.1.2:32001 -P -t test
```

--------------------------------

### List Docker Services

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Displays a list of all services currently running in the Docker Swarm cluster, showing their names, replicas, and image.

```bash
docker service ls
```

--------------------------------

### Kafka Configuration: Current Listener Settings

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Demonstrates the current environment variables for configuring Kafka listeners, supporting multiple protocols and advertised addresses.

```APIDOC
Kafka Listener Configuration (Current):

KAFKA_LISTENERS: Defines the listener(s) the broker binds to, specifying protocol and port. Supports multiple listeners.
KAFKA_ADVERTISED_LISTENERS: Defines the listener(s) clients should use to connect, specifying protocol and advertised host/port. Supports multiple listeners.

Example:
KAFKA_LISTENERS: PLAINTEXT://:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://one.prod.com:9092
```

--------------------------------

### Connect to Running Kafka Docker Container

Source: https://github.com/wurstmeister/kafka-docker/wiki/Home

Demonstrates how to access a running Kafka Docker container using the 'docker exec' command for interactive sessions.

```shell
docker exec -it kafkadocker_kafka_1
```

--------------------------------

### Create Docker Machine Nodes

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Commands to create new Docker machine instances, typically used for setting up swarm nodes. Replace `<name>` with your desired node name (e.g., 'manager', 'worker').

```bash
docker-machine create <name>

# i.e.
docker-machine create manager
docker-machine create worker
```

--------------------------------

### List Kafka Docker Tags

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Fetches and lists all available tags for the wurstmeister/kafka Docker image from Docker Hub. This command uses curl to query the Docker Hub API, jq to parse the JSON response, and sort/egrep for filtering and ordering.

```bash
curl -s https://registry.hub.docker.com/v2/repositories/wurstmeister/kafka/tags?page_size=1024 | jq -r '.results[].name' | sort -u | egrep '\d.\d{2}-.*'
```

--------------------------------

### Configure Docker Compose Windows Path Conversion

Source: https://github.com/wurstmeister/kafka-docker/wiki/Home

Addresses issues with mounting \\var\\run\\docker.sock on Windows by setting the COMPOSE_CONVERT_WINDOWS_PATHS environment variable.

```shell
echo "COMPOSE_CONVERT_WINDOWS_PATH=1" >> .env
```

--------------------------------

### Run Selected Kafkacat Tests

Source: https://github.com/wurstmeister/kafka-docker/blob/master/test/Readme.md

This command executes specific Kafkacat tests. It first retrieves the broker list, optionally sets the Kafka version, and then runs the tests using `docker-compose run` within the `kafkacattest` container. The `<testname pattern>` can be a filename or a wildcard.

```shell
BROKER_LIST=$(./internal-broker-list.sh) [KAFKA_VERSION=<version>] docker-compose run --rm kafkacattest <testname pattern>
```

--------------------------------

### Kafka Configuration via Environment Variables

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Explains how to customize Kafka broker configurations by setting environment variables in the docker-compose.yml file. This covers general Kafka parameters and log4j settings.

```APIDOC
KAFKA_ADVERTISED_HOST_NAME: The IP address of the Docker host. Must be set to match your environment.

KAFKA_MESSAGE_MAX_BYTES: 2000000
  - Sets the Kafka broker's message max bytes configuration.

KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
  - Disables automatic topic creation.

LOG4J_LOGGER_KAFKA_AUTHORIZER_LOGGER: DEBUG, authorizerAppender
  - Customizes Kafka's log4j settings, specifically for the authorizer logger.
```

--------------------------------

### Configure Kafka Listener Configuration

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Manages Kafka broker listener configurations, supporting multiple protocols and discriminating between internal and external traffic. Deprecates `advertised.host.name` and `advertised.port` in favor of listener configurations.

```APIDOC
KAFKA_LISTENERS: "<listener_name>://:<port>[,<listener_name>://:<port>]..."
  - Defines the network interfaces and ports the Kafka broker listens on.
  - Example: "INSIDE://:9092,OUTSIDE://:9094"

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "<listener_name>:<protocol>[,<listener_name>:<protocol>]..."
  - Maps listener names to security protocols (e.g., PLAINTEXT, SSL, SASL_SSL).
  - Example: "INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT"

KAFKA_INTER_BROKER_LISTENER_NAME: "<listener_name>"
  - Specifies the listener name used for inter-broker communication.
  - Example: "INSIDE"

KAFKA_ADVERTISED_LISTENERS: "<listener_name>://[<hostname>]:<port>[,<listener_name>://[<hostname>]:<port>]..."
  - Defines how clients should connect to the broker, potentially using dynamic hostnames or ports.
  - Can interpolate values from HOSTNAME_COMMAND or PORT_COMMAND using `_{VAR_NAME}` syntax.
  - Example: "INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094"

Rules:
  - No listeners may share a port number.
  - An advertised.listener must be present by protocol name and port number in the list of listeners.
```

--------------------------------

### Initialize Docker Swarm Manager

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Initializes a new Docker Swarm on the current node, designating it as the manager. It provides the join token and address needed for other nodes to join the swarm.

```bash
docker-machine ssh manager
docker@manager:~$ docker swarm init --advertise-addr=192.168.99.100

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-3fzauu1a999bid7z7rvteujkfedg9t5fbzr3ivjanuaroxs6vm-4gtl6ban1x2nvad6wsucmlxgv 192.168.99.100:2377
```

--------------------------------

### Kafka Configuration: Deprecated Listener Settings

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Shows the deprecated environment variables for configuring Kafka host and port, including advertised settings.

```APIDOC
Kafka Listener Configuration (Deprecated):

KAFKA_HOST: Specifies the internal hostname for the Kafka broker.
KAFKA_PORT: Specifies the internal port for the Kafka broker.
KAFKA_ADVERTISED_HOST_NAME: Specifies the external hostname clients should use to connect.
KAFKA_ADVERTISED_PORT: Specifies the external port clients should use to connect.

Example:
KAFKA_HOST: 
KAFKA_PORT: 9092
KAFKA_ADVERTISED_HOST_NAME: one.prod.com
KAFKA_ADVERTISED_PORT: 9092
```

--------------------------------

### Connect to Kafka JMX with Jconsole

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Command to connect to a Kafka broker's JMX interface using the Jconsole tool. Assumes JMX is configured and accessible on the specified host and port.

```shell
jconsole 192.168.99.100:1099
```

--------------------------------

### Kafka Connectivity Test with Kafkacat

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Tests connectivity to Kafka brokers running in Docker Swarm using `kafkacat`. It demonstrates producing messages to a topic and consuming messages from it.

```bash
# /etc/hosts
192.168.99.100  manager
192.168.99.101  worker

# Produce messages
$ kafkacat -b manager:9094,worker:9094 -P -t test
foo
bar
^C

# Consume messages
$ kafkacat -b manager:9094,worker:9094 -C -t test
foo
bar
^C
```

--------------------------------

### Configure Advertised Port

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Sets the advertised port for Kafka brokers, allowing dynamic resolution via shell commands. This is crucial when the port is not static.

```APIDOC
PORT_COMMAND: "<command>"
  - Executes a shell command to dynamically determine the advertised port.
  - The output of the command is used as the port number.
  - Example: "docker port $$(hostname) 9092/tcp | cut -d: -f2" to get the mapped TCP port.
```

--------------------------------

### Injecting Hostname Command into Configuration

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Demonstrates how to use the output of `HOSTNAME_COMMAND` within other Kafka configuration variables using a special interpolation syntax.

```yaml
environment:
  HOSTNAME_COMMAND: "curl http://169.254.169.254/latest/meta-data/public-hostname"
  KAFKA_ADVERTISED_LISTENERS: "SSL://_{HOSTNAME_COMMAND}:9093,PLAINTEXT://9092"
```

--------------------------------

### Configure Advertised Hostname

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Sets the advertised hostname for Kafka brokers, allowing explicit configuration or dynamic resolution via shell commands. The explicit hostname takes precedence.

```APIDOC
KAFKA_ADVERTISED_HOST_NAME: "<hostname>"
  - Explicitly sets the advertised hostname for the Kafka broker.
  - Takes precedence over HOSTNAME_COMMAND.

HOSTNAME_COMMAND: "<command>"
  - Executes a shell command to dynamically determine the advertised hostname.
  - The output of the command is used as the hostname.
  - Example: "route -n | awk '/UG[ \t]/{print $$2}'" to get the default gateway IP.
  - Example for AWS: "wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4"
```

--------------------------------

### Kafka Broker Registration Metadata Structure

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Illustrates the JSON structure Kafka brokers register with ZooKeeper, containing listener information, host, port, and version.

```json
{
  "listener_security_protocol_map": {
    "PLAINTEXT": "PLAINTEXT"
  },
  "endpoints": [
    "PLAINTEXT://one.prod.com:9092"
  ],
  "jmx_port": -1,
  "host": "one.prod.com",
  "timestamp": "1520972878198",
  "port": 9092,
  "version": 4
}
```

--------------------------------

### Configure Broker Rack Affinity

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Sets the Kafka broker's rack affinity, allowing explicit configuration or dynamic resolution using shell commands, often leveraging cloud provider metadata services.

```APIDOC
KAFKA_BROKER_RACK: "<rack_id>"
  - Explicitly sets the rack ID for the Kafka broker.

RACK_COMMAND: "<command>"
  - Executes a shell command to dynamically determine the broker's rack ID.
  - The output of the command is used as the rack ID.
  - Example for AWS: "curl http://169.254.169.254/latest/meta-data/placement/availability-zone"
```

--------------------------------

### Kafka Broker ID Configuration

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Details methods for configuring Kafka broker IDs, which is crucial for managing cluster state and ensuring consistent broker identification, especially when scaling or restarting.

```APIDOC
KAFKA_BROKER_ID: <integer>
  - Explicitly sets the broker ID for a Kafka instance.

BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $$2}'"
  - Specifies a command to dynamically determine the broker ID, often based on the container's hostname.
```

--------------------------------

### Run Selected Kafka Tests

Source: https://github.com/wurstmeister/kafka-docker/blob/master/test/Readme.md

This command allows running specific Kafka tests by providing a pattern. It uses `docker-compose run` to execute the tests within the `kafkatest` container, ensuring isolation and clean execution.

```shell
docker-compose run --rm kafkatest <testname pattern>
```

--------------------------------

### Execute JMX Scenario Test

Source: https://github.com/wurstmeister/kafka-docker/blob/master/test/scenarios/Readme.md

This snippet demonstrates how to execute the JMX scenario test for kafka-docker. It involves navigating to the test directory and running a specific shell script. This test verifies the JMX configuration.

```shell
cd test/scenarios
./runJmxScenario.sh
```

--------------------------------

### Switch Docker Context

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Activates the environment for a specific Docker machine, allowing subsequent Docker commands to target that machine. This is essential before performing swarm operations.

```bash
eval $(docker-machine env <name>)
```

--------------------------------

### Injecting Port Command into Configuration

Source: https://github.com/wurstmeister/kafka-docker/blob/master/README.md

Shows how to use the output of `PORT_COMMAND` within other Kafka configuration variables via interpolation.

```yaml
environment:
  PORT_COMMAND: "docker port $$(hostname) 9092/tcp | cut -d: -f2"
  KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://1.2.3.4:_{PORT_COMMAND}"
```

--------------------------------

### Deploy Docker Swarm Stack

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Deploys a stack defined in a Docker Compose file to the Swarm cluster. This command should be run on the manager node.

```bash
docker-machine ssh manager
docker stack deploy -c docker-compose-swarm.yml kafka
```

--------------------------------

### Kafka Broker Environment Variables

Source: https://github.com/wurstmeister/kafka-docker/wiki/SASL-config-example

Configures Kafka broker settings for SASL authentication, network listeners, replication factors, and topic creation. It leverages Ansible variables for dynamic configuration, including hostnames and availability zones.

```yaml
env:
  - "BROKER_ID_COMMAND='hostname | cut -b 20'"
  - "KAFKA_ADVERTISED_HOST_NAME={{ ansible_hostname }}.node.consul.{{internalDNS}}."
  - "RACK_COMMAND='curl http://169.254.169.254/latest/meta-data/placement/availability-zone'"
  - "KAFKA_ZOOKEEPER_CONNECT={{ec2_tag_cluster}}.zookeeper.service.consul.{{internalDNS}}.:2181"
  - "KAFKA_DEFAULT_REPLICATION_FACTOR=3"
  - "KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3"
  - "KAFKA_CREATE_TOPICS={% for queue in queues %}{{ queue }}.topic:10:2,{% endfor %}__consumer_offsets:50:3"
  - "KAFKA_MESSAGE_MAX_BYTES=256000"
  - "KAFKA_AUTO_CREATE_TOPICS_ENABLE=false"
  - "KAFKA_LOG_DIRS=/logs/kafka"
  - "KAFKA_SASL_ENABLED_MECHANISMS=SCRAM-SHA-256,SCRAM-SHA-512"
  - "KAFKA_LISTENERS=SASL_PLAINTEXT://:9092"
  - "KAFKA_ADVERTISED_LISTENERS=SASL_PLAINTEXT://{{ ansible_hostname }}.node.consul.{{internalDNS}}.:9092"
  - "KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=SASL_PLAINTEXT:SASL_PLAINTEXT"
  - "KAFKA_INTER_BROKER_LISTENER_NAME=SASL_PLAINTEXT"
  - "KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-256"
  - "KAFKA_AUTHORIZER_CLASS_NAME=kafka.security.authorizer.AclAuthorizer"
  - "KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true"
  - "KAFKA_SUPER_USERS=User:admin"
  - "KAFKA_OPTS=-Djava.security.auth.login.config=/app/config/jaas.conf"
```

--------------------------------

### Join Docker Swarm as Worker

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Connects a Docker node to an existing Swarm cluster as a worker. Requires the manager's IP address and the join token obtained from the `docker swarm init` command.

```bash
docker-machine ssh worker
docker@worker:~$ docker swarm join --token <token> 192.168.99.100:2377
```

--------------------------------

### Scale Docker Service

Source: https://github.com/wurstmeister/kafka-docker/wiki/Swarm

Adjusts the number of running instances (replicas) for a specific service within the Swarm. Scaling is limited by the number of available nodes and port configurations.

```bash
docker service scale kafka_kafka=2
```

--------------------------------

### Create Kafka Consumer ACL using kafka-acls.sh

Source: https://github.com/wurstmeister/kafka-docker/wiki/SASL-config-example

This Ansible task demonstrates how to create an Access Control List (ACL) for a Kafka consumer. It utilizes the kafka-acls.sh script to grant specific permissions to a user principal for a given topic and consumer group. The task includes conditional execution based on user type and registers the command's output for potential further processing or validation.

```ansible
- name: "Create {{user.name}} consumer acl for {{user.topic}}"
  command: >
    /usr/local/sbin/kafka-cmd kafka-acls.sh \
    --authorizer-properties zookeeper.connect={{kafka.config.zookeeper.address}} \
    --add \
    --allow-principal 'User:{{ user.name }}' \
    --consumer \
    --group '{{ user.group }}' \
    --topic '{{ user.topic }}'
  register: command_result
  when: 'user.type == "consumer" or user.type =="both" '
  failed_when:
    - 'user.name ~ " has Allow permission for operations: Create from hosts:" not in command_result.stdout'
```

--------------------------------

### Fix Kafka Tools Failure with JMX Enabled

Source: https://github.com/wurstmeister/kafka-docker/wiki/Home

When JMX is enabled, Kafka tools like kafka-topics.sh can fail due to port conflicts. This snippet shows how to resolve this by prefixing commands with 'JMX_PORT=' or unsetting the variable.

```shell
JMX_PORT= ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
```

```shell
unset JMX_PORT
```

--------------------------------

### Kafka Broker Port Configuration for Multiple Brokers

Source: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

Shows the docker-compose configuration for exposing Kafka broker ports when running multiple brokers. Instead of mapping to specific host ports, it exposes the container port, allowing Docker to assign ephemeral host ports.

```docker-compose
ports:
  - "9092"
```

--------------------------------

### Kafka User and ACL Management

Source: https://github.com/wurstmeister/kafka-docker/wiki/SASL-config-example

Ansible tasks to manage Kafka users and Access Control Lists (ACLs) using Kafka's command-line tools. This includes creating users with SCRAM credentials and defining producer/consumer ACLs for specific topics and groups.

```APIDOC
Kafka User and ACL Management:

1. Create Admin Authentication Credentials in ZooKeeper:
   - Command: kafka-configs.sh --zookeeper <zookeeper_address> --alter --add-config 'SCRAM-SHA-256=[password=<password>],SCRAM-SHA-512=[password=<password>]' --entity-type users --entity-name admin
   - Purpose: Adds SCRAM-SHA-256 and SCRAM-SHA-512 credentials for the 'admin' user.
   - Parameters:
     - --zookeeper: Address of the ZooKeeper ensemble.
     - --alter: Modifies existing configuration.
     - --add-config: Specifies the configuration to add.
     - --entity-type: Type of entity to configure (e.g., 'users').
     - --entity-name: Name of the entity (e.g., 'admin').
   - Returns: Confirmation message upon successful update.
   - Example Usage (Ansible):
     command: >
       /usr/local/sbin/kafka-cmd kafka-configs.sh \
       --zookeeper {{kafka.config.zookeeper.address}} \
       --alter \
       --add-config 'SCRAM-SHA-256=[password={{ lookup('vault', 'secret/kafka/admin', 'value') }}],SCRAM-SHA-512=[password={{ lookup('vault', 'secret/kafka/admin', 'value') }}]' \
       --entity-type users \
       --entity-name admin
   - Error Handling: Fails if 'Completed Updating config for entity: user-principal' is not in stdout.

2. Deny Anonymous Access ACL:
   - Command: kafka-acls.sh --authorizer-properties zookeeper.connect=<zookeeper_address> --add --cluster --operation Alter --deny-principal User:ANONYMOUS
   - Purpose: Denies the 'Alter' cluster operation for the anonymous user.
   - Parameters:
     - --authorizer-properties: Properties for the authorizer, including ZooKeeper connection.
     - --add: Adds a new ACL entry.
     - --cluster: Applies the ACL to the cluster resource.
     - --operation: The operation to control (e.g., 'Alter').
     - --deny-principal: The principal to deny permissions to.
   - Returns: Confirmation message upon successful ACL addition.
   - Example Usage (Ansible):
     command: >
       /usr/local/sbin/kafka-cmd kafka-acls.sh \
       --authorizer-properties zookeeper.connect={{kafka.config.zookeeper.address}} \
       --add \
       --cluster \
       --operation Alter \
       --deny-principal User:ANONYMOUS
   - Error Handling: Fails if 'User:ANONYMOUS has Deny permission for operations: Alter' is not in stdout.

3. Create User Authentication Credentials:
   - Command: kafka-configs.sh --zookeeper <zookeeper_address> --alter --add-config 'SCRAM-SHA-256=[password=<password>],SCRAM-SHA-512=[password=<password>]' --entity-type users --entity-name <user_name>
   - Purpose: Adds SCRAM credentials for a specified user.
   - Parameters:
     - --zookeeper: Address of the ZooKeeper ensemble.
     - --alter: Modifies existing configuration.
     - --add-config: Specifies the configuration to add.
     - --entity-type: Type of entity to configure ('users').
     - --entity-name: Name of the user.
   - Returns: Confirmation message upon successful update.
   - Example Usage (Ansible):
     command: >
       /usr/local/sbin/kafka-cmd kafka-configs.sh \
       --zookeeper {{kafka.config.zookeeper.address}} \
       --alter \
       --add-config 'SCRAM-SHA-256=[password={{ user.password }}],SCRAM-SHA-512=[password={{ user.password }}]' \
       --entity-type users \
       --entity-name {{ user.name }}
   - Error Handling: Fails if the expected success message is not found.

4. Create User Producer ACL:
   - Command: kafka-acls.sh --authorizer-properties zookeeper.connect=<zookeeper_address> --add --allow-principal 'User:<user_name>' --producer --group '<group_name>' --topic '<topic_name>'
   - Purpose: Grants producer access to a specific topic for a user and group.
   - Parameters:
     - --authorizer-properties: ZooKeeper connection for the authorizer.
     - --add: Adds a new ACL entry.
     - --allow-principal: The principal to grant permissions to.
     - --producer: Specifies producer access.
     - --group: The consumer group associated with the ACL.
     - --topic: The topic the ACL applies to.
   - Returns: Confirmation message upon successful ACL addition.
   - Example Usage (Ansible):
     command: >
       /usr/local/sbin/kafka-cmd kafka-acls.sh \
       --authorizer-properties zookeeper.connect={{kafka.config.zookeeper.address}} \
       --add \
       --allow-principal 'User:{{ user.name }}' \
       --producer \
       --group '{{ user.group }}' \
       --topic '{{ user.topic }}'
   - Conditions: Executes when user.type is 'producer' or 'both'.
   - Error Handling: Fails if the ACL creation command encounters an error.

User List Format:
  kafka_users:
    - name: user
      password:  pass
      topic: topic
      group: consumer_group
      type: (both|consumer|producer)
```

--------------------------------

### Kafka JAAS Configuration

Source: https://github.com/wurstmeister/kafka-docker/wiki/SASL-config-example

Defines the Java Authentication and Authorization Service (JAAS) configuration for Kafka, specifying the SCRAM login module for server and client authentication. It includes placeholders for admin credentials retrieved from a vault.

```kafka-config
KafkaServer {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="admin"
  password="{{ lookup('vault', 'secret/kafka/admin', 'value') }}";
};
KafkaClient {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="admin"
  password="{{ lookup('vault', 'secret/kafka/admin', 'value') }}";
};
Client {};
```
Tokens:
Raw