### Start ksqlDB Server Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/how-it-works.md Use this command to start a ksqlDB server instance. Ensure you replace `` with the actual path to your Confluent installation. ```bash /bin/ksql-server-start ``` -------------------------------- ### PRINT FROM BEGINNING Example Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/appendix.md Retrieve all messages from the start of a topic using the BEGINNING keyword. ```sql PRINT FROM BEGINNING; ``` -------------------------------- ### Start ksqlDB Server with Configuration File Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/server-config/index.md Command to start the ksqlDB Server using a specified configuration file. ```bash /bin/ksql-server-start /etc/ksqldb/ksql-server.properties ``` -------------------------------- ### Install conventional-changelog-cli Source: https://github.com/confluentinc/ksql/blob/master/scripts/changelog/README.md Install the command-line interface tool used for generating changelogs. This is a one-time setup step. ```bash npm install -g conventional-changelog-cli ``` -------------------------------- ### Start ksqlDB Server and CLI Source: https://github.com/confluentinc/ksql/blob/master/CONTRIBUTING.md Start the ksqlDB server in the background and the ksqlDB CLI in the foreground. Ensure Kafka and Schema Registry are running beforehand. ```shell $ ./bin/ksql-server-start -daemon config/ksql-server.properties $ ./bin/ksql ``` -------------------------------- ### Create Stream Example with PARTITIONS and REPLICAS Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-2-produce-data.md Example of creating a stream where the underlying Kafka topic is created with specific partition and replica counts. ```sql CREATE STREAM BAR (f0, f1) WITH (Kafka_topic='foo', PARTITIONS=4, REPLICAS=2); ``` -------------------------------- ### Example ksqlDB Server Docker Logs Output Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/install-ksqldb-with-docker.md This is an example of the log output you might see when viewing ksqlDB Server logs. It shows the start of a stream thread. ```text [2019-01-16 23:43:05,591] INFO stream-thread [_confluent-ksql-default_transient_1507119262168861890_1527205385485-71c8a94c-abe9-45ba-91f5-69a762ec5c1d-StreamThread-17] Starting (org.apache.kafka.streams.processor.internals.StreamThread:713) ... ``` -------------------------------- ### Install MkDocs Source: https://github.com/confluentinc/ksql/blob/master/docs/README.md Installs the MkDocs static site generator using pip. Ensure Python 3.7+ and pip are installed. ```bash pip install mkdocs ``` -------------------------------- ### SHOW CONNECTORS Example Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/appendix.md List all available connectors using the SHOW CONNECTORS command. ```sql SHOW CONNECTORS; ``` -------------------------------- ### Successful Migration Output Example Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/migrations-tool.md Example output after a successful migration, confirming that the migration statements were applied to the ksqlDB cluster. ```text Validating current migration state before applying new migrations Loading migration files 1 migration file(s) loaded. Applying migration version 2: Add users /my/migrations/project/migrations/V000002__Add_users.sql contents: CREATE OR REPLACE TABLE users (user_id VARCHAR PRIMARY KEY, email VARCHAR, country VARCHAR) WITH (KAFKA_TOPIC='users', FORMAT='JSON'); Successfully migrated Execution time: 1.2320 seconds ``` -------------------------------- ### Install ksqlDB .NET LINQ Provider Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-51-ksqldb .NET LINQ provider.md Install the Kafka.DotNet.ksqlDB NuGet package using the Package Manager Console. ```powershell Install-Package Kafka.DotNet.ksqlDB ``` -------------------------------- ### Example ksqlDB DESCRIBE CONNECTOR Usage Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-reference/describe-connector.md This example demonstrates how to use the DESCRIBE CONNECTOR statement to get details about a connector named 'my-jdbc-connector'. It will show the connector's status and any ksqlDB objects it has created. ```sql DESCRIBE CONNECTOR "my-jdbc-connector"; ``` -------------------------------- ### KSQL Stream Creation Examples with Wrapping Options Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-3-serialization-of-single-fields.md Illustrates various KSQL stream creation scenarios, demonstrating the use of system defaults and explicit WRAP_SINGLE_VALUE/WRAP_SINGLE_KEY properties for controlling key and value serialization. Includes examples of valid and erroneous configurations. ```sql -- Assuming system configuration is at the default: -- ksql.persistence.wrap.single.keys=false -- ksql.persistence.wrap.single.values=true -- creates a stream, picking up the system default of not wrapping keys and wrapping values. -- the serialized key is expected to not be wrapped. -- the serialized value is expected to be wrapped. -- if the serialized forms do not match the expected wrapping it will result in a deserialization error. CREATE STREAM IMPLICIT_SOURCE (ID INT KEY, NAME STRING) WITH (...); -- override 'ksql.persistence.wrap.single.values' to false -- the serialized value is expected to not be unwrapped. CREATE STREAM EXPLICIT_SOURCE (ID INT) WITH (WRAP_SINGLE_VALUE=false, ...); -- results in an error as the value schema is multi-field CREATE STREAM BAD_SOURCE (ID INT, NAME STRING) WITH (WRAP_SINGLE_VALUE=false, ...); -- creates a stream, picking up the system default of not wrapping keys and wrapping values. -- the serialized keys in the sink topic will not be wrapped. -- the serialized values in the sink topic will be wrapped. CREATE STREAM IMPLICIT_SINK AS SELECT ID FROM S; -- override 'ksql.persistence.wrap.single.keys' to true -- the serialized keys and values will be wrapped. CREATE STREAM EXPLICIT_SINK WITH(WRAP_SINGLE_KEY=true) AS SELECT ID FROM S; -- results in an error as the value schema is multi-field CREATE STREAM BAD_SINK WITH(WRAP_SINGLE_VALUE=true) AS SELECT ID, COST FROM S; ``` -------------------------------- ### GROUP BY, ADVANCE BY, and PARTITION BY Examples Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/appendix.md Demonstrates the usage of BY for grouping, window advancement, and partitioning. ```sql GROUP BY regionid ``` ```sql ADVANCE BY 10 SECONDS ``` ```sql PARTITION BY userid ``` -------------------------------- ### Install MkDocs Plugins and Extensions Source: https://github.com/confluentinc/ksql/blob/master/docs/README.md Installs required MkDocs plugins and extensions listed in the requirements.txt file using pip. This command should be run from the root of the ksqlDB repository. ```bash pip install -r docs/requirements.txt ``` -------------------------------- ### Example Password File Content Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/server-config/security.md An example of a password file demonstrating different users, their encrypted passwords (OBF, MD5, CRYPT), and assigned roles. ```properties fred: OBF:1w8t1tvf1w261w8v1w1c1tvn1w8x,user,admin harry: changeme,user,developer tom: MD5:164c88b302622e17050af52c89945d44,user dick: CRYPT:adpexzg3FUZAk,admin,ksql-user ``` -------------------------------- ### Install Postgres Debezium Connector Source: https://github.com/confluentinc/ksql/blob/master/docs/tutorials/etl.md Download and install the Debezium connector for PostgreSQL using the confluent-hub client. ```bash confluent-hub install --component-dir confluent-hub-components --no-prompt debezium/debezium-connector-postgresql:1.1.0 ``` -------------------------------- ### Install ksqlDB UDF/UDAF Archetype Locally Source: https://github.com/confluentinc/ksql/blob/master/ksqldb-udf-quickstart/README.md Install the archetype to your local Maven repository for testing purposes. This allows you to test changes before deploying. ```bash $ mvn install ``` -------------------------------- ### Install npm for Commitlint Source: https://github.com/confluentinc/ksql/blob/master/CONTRIBUTING.md Install npm from the root directory of the ksqlDB repo to enable commitlint. This tool enforces the conventional commit message format. ```bash npm install ``` -------------------------------- ### CREATE SOURCE CONNECTOR Example Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/appendix.md Define and create a new source connector with specified properties. ```sql CREATE SOURCE CONNECTOR 'jdbc-connector' WITH( … ``` -------------------------------- ### Start Headless ksqlDB Server with Queries File Argument Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/server-config/index.md Use this command-line argument to start ksqlDB Server in headless, non-interactive mode with a predefined SQL script. ```bash /bin/ksql-server-start /etc/ksqldb/ksql-server.properties \ --queries-file /path/to/queries.sql ``` -------------------------------- ### Dry Run Output Example Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/migrations-tool.md Example output from a dry run, showing the migration file that would be applied and its contents, without executing any changes. ```text This is a dry run. No ksqlDB statements will be submitted to the ksqlDB server. Validating current migration state before applying new migrations Loading migration files 1 migration file(s) loaded. Applying migration version 2: Add users /my/migrations/project/migrations/V000002__Add_users.sql contents: CREATE OR REPLACE TABLE users (user_id VARCHAR PRIMARY KEY, email VARCHAR, country VARCHAR) WITH (KAFKA_TOPIC='users', FORMAT='JSON'); Dry run complete. No migrations were actually applied. Execution time: 1.2270 seconds ``` -------------------------------- ### Install a Connector using Confluent Hub CLI Source: https://github.com/confluentinc/ksql/blob/master/docs/how-to-guides/use-connector-management.md Download and install a connector package using the `confluent-hub` utility. Specify the component directory and the connector name with version. ```bash confluent-hub install --component-dir confluent-hub-components --no-prompt mdrogalis/voluble:{{ site.voluble_version }} ``` -------------------------------- ### Start the ksqlDB Docker Compose stack Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/installing.md Use this command to start all ksqlDB services in detached mode. The initial startup may take several minutes. ```bash docker-compose up -d ``` -------------------------------- ### Basic ksqlDB Server Properties Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/server-config/index.md Example of basic properties for ksqlDB Server configuration file. ```properties bootstrap.servers=localhost:9092 listeners=http://localhost:8088 ``` -------------------------------- ### Start ksqlDB Node in Headless Mode Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/how-it-works.md Use this command to start a ksqlDB node in headless mode and specify a SQL file for queries. Ensure the path to the ksql-node executable and the SQL file are correct. ```bash bin/ksql-node query-file=path/to/myquery.sql ``` -------------------------------- ### SHOW ALL TOPICS Example Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/appendix.md Use the SHOW ALL TOPICS command to list all hidden topics. ```sql SHOW ALL TOPICS; ``` -------------------------------- ### Initialize ksqlDB Java Client Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-clients/java-client.md This example demonstrates how to create a ksqlDB client instance using `ClientOptions`. Ensure the host and port are correctly configured for your ksqlDB server. ```java package my.ksqldb.app; import io.confluent.ksql.api.client.Client; import io.confluent.ksql.api.client.ClientOptions; public class ExampleApp { public static String KSQLDB_SERVER_HOST = "localhost"; public static int KSQLDB_SERVER_HOST_PORT = 8088; public static void main(String[] args) { ClientOptions options = ClientOptions.create() .setHost(KSQLDB_SERVER_HOST) .setPort(KSQLDB_SERVER_HOST_PORT); Client client = Client.create(options); // Send requests with the client by following the other examples // Terminate any open connections and close the client client.close(); } } ``` -------------------------------- ### Build and Serve ksqlDB Docs Locally Source: https://github.com/confluentinc/ksql/blob/master/docs/README.md Builds the ksqlDB documentation and starts a local development server. Open http://127.0.0.1:8000 in a web browser to view the docs. The server automatically rebuilds on file changes. ```bash mkdocs serve ``` -------------------------------- ### DESCRIBE Example Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/appendix.md Retrieve detailed information about a specific object like a stream or table. ```sql DESCRIBE PAGEVIEWS; ``` -------------------------------- ### Start ksqlDB CLI with a Configuration File in Docker Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/install-ksqldb-with-docker.md This command launches the ksqlDB CLI in a container, connecting to a ksqlDB Server and using a specified configuration file mounted from the host. Ensure the configuration file exists on the host. ```bash # Assume ksqlDB Server is running. # Ensure that the configuration file exists. ls /path/on/host/ksql-cli.properties docker run -it \ -v /path/on/host/:/path/in/container \ confluentinc/ksqldb-cli:{{ site.ksqldbversion }} ksql http://10.0.0.11:8088 \ --config-file /path/in/container/ksql-cli.properties ``` -------------------------------- ### CHR Function Examples Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-reference/scalar-functions.md Demonstrates using the CHR function with decimal codes and UTF-8 string representations to get characters. ```sql CHR(75) => 'K' CHR('\u004b') => 'K' CHR(22909) => '好' CHR('\u597d') => '好' ``` -------------------------------- ### Kafka Record Example Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-2-produce-data.md Illustrates the expected structure of Kafka records for a KSQL table. ```json {"key": "bar", "timestamp": 1, "value": {"id": "bar", "foo": 123}} ``` -------------------------------- ### Install Elasticsearch Connector Source: https://github.com/confluentinc/ksql/blob/master/docs/tutorials/etl.md Download and install the Elasticsearch connector for Kafka using the confluent-hub client. ```bash confluent-hub install --component-dir confluent-hub-components --no-prompt confluentinc/kafka-connect-elasticsearch:10.0.1 ``` -------------------------------- ### Install MongoDB Debezium Connector Source: https://github.com/confluentinc/ksql/blob/master/docs/tutorials/etl.md Download and install the Debezium connector for MongoDB using the confluent-hub client. ```bash confluent-hub install --component-dir confluent-hub-components --no-prompt debezium/debezium-connector-mongodb:1.1.0 ``` -------------------------------- ### AVRO Physical Schema Example Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/schema-registry-integration.md An example of a physical schema in AVRO format registered with Schema Registry. ```json { "schema": { "type": "record", "name": "PageViewValueSchema", "namespace": "io.confluent.ksql.avro_schemas", "fields": [ { "name": "page_name", "type": "string", "default": "abc" }, { "name": "ts", "type": "int", "default": 123 } ] } } ``` -------------------------------- ### Setting Server/CLI Configurations with Variables Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-38-variable-substitution.md Illustrates how to use variables to set server or CLI configurations, such as 'auto.offset.reset' and CLI wrap toggle. ```sql ksql> DEFINE offset = 'earliest' ksql> SET 'auto.offset.reset' = '${offset}'; ksql> DEFINE wrap_toggle = 'ON' ksql> SET CLI WRAP ${WRAP_TOGGLE} ``` -------------------------------- ### Connect to the ksqlDB CLI Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/installing.md Execute this command to launch the ksqlDB CLI and connect to the primary ksqlDB server. Ensure all services are in the 'Up' state before running. ```bash docker exec -it ksqldb-cli ksql http://primary-ksqldb-server:8088 ``` -------------------------------- ### Insert Data for Filter Example Source: https://github.com/confluentinc/ksql/blob/master/docs/how-to-guides/use-lambda-functions.md Inserts sample data into the stream created for the FILTER lambda function example. ```sql INSERT INTO stream3 ( id, lambda_map ) VALUES ( 1, MAP('first name':= 15, 'middle':= 25, 'last name':= 0, 'alt name':= 33) ); ``` -------------------------------- ### Insert Data for Reduce Example Source: https://github.com/confluentinc/ksql/blob/master/docs/how-to-guides/use-lambda-functions.md Inserts sample data into the stream created for the REDUCE lambda function example. ```sql INSERT INTO stream2 ( id, lambda_arr ) VALUES ( 1, ARRAY[2, 3, 4, 5] ); ``` -------------------------------- ### Show All Connectors Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-7-connect-integration.md Lists all configured connectors, their classes, and topic prefixes. Useful for verifying connector setup. ```sql ksql> SHOW CONNECTORS; name | connector | topic prefix -----------------|-----------------------------------------------|-------------------- my-postgres-jdbc | io.confluent.connect.jdbc.JdbcSourceConnector | test-postgres-jdbc ``` -------------------------------- ### Launch ksqlDB CLI Source: https://github.com/confluentinc/ksql/blob/master/docs/how-to-guides/use-connector-management.md Connect to the ksqlDB CLI to interact with the ksqlDB server. This is the first step to managing connectors. ```bash docker exec -it ksqldb-cli ksql http://ksqldb-server:8088 ``` -------------------------------- ### External Schema Example Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-5-internal-schemas-for-KSQL-engine.md An example of an external schema with non-KSQL-compliant field names, including quoted identifiers and special characters. ```sql ("@ID" BIGINT, "@NAME" STRING, "MESSAGE.AMOUNT" DOUBLE, "SELECT" STRING) ``` -------------------------------- ### Run Interactive ksqlDB Server with Docker Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/install-ksqldb-with-docker.md Start a ksqlDB Server container for interactive development using the ksqlDB CLI. Key settings include bootstrap servers, listeners, and a service ID. ```bash docker run -d \ -p 127.0.0.1:8088:8088 \ -e KSQL_BOOTSTRAP_SERVERS=localhost:9092 \ -e KSQL_LISTENERS=http://0.0.0.0:8088/ \ -e KSQL_KSQL_SERVICE_ID=ksql_service_2_ \ confluentinc/ksqldb-server:{{ site.ksqldbversion }} ``` -------------------------------- ### Initial ksqlDB Schema Setup Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-42-schema-migrations-tool.md This SQL script defines the initial state of a ksqlDB cluster, including a stream for pageviews and a table for metrics derived from it. It's intended to be used as the first migration file. ```sql CREATE STREAM pageviews ( user_id INTEGER KEY, url STRING, status INTEGER ) WITH ( KAFKA_TOPIC='pageviews', VALUE_FORMAT='JSON' ); CREATE TABLE pageviews_metrics AS SELECT url, COUNT(*) AS num_views FROM pageviews GROUP BY url EMIT CHANGES; ``` -------------------------------- ### Null Value Directive Example Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-32-sql-testing-tool.md Example directive to produce a null-valued record into a specific topic/source, for supporting null values in streams. ```sql --@null.value: topic key ``` -------------------------------- ### Configure Headless ksqlDB Server via Configuration File Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/server-config/index.md Set `ksql.queries.file` in the server properties file to start ksqlDB Server in headless, non-interactive mode. Ensure `bootstrap.servers` is also configured. ```properties # Inform the ksqlDB server where the Kafka cluster can be found: bootstrap.servers=localhost:9092 # Define the location of the queries file to execute ksql.queries.file=/path/to/queries.sql ``` -------------------------------- ### Create a New ksql-migrations Project Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/migrations-tool.md Initializes a new ksql-migrations project with the specified directory structure and configuration file. Requires the project path and ksqlDB server URL. ```bash ksql-migrations new-project [--] ``` ```bash $ ksql-migrations new-project /my/migrations/project/path http://localhost:8088 ``` ```text Creating new migrations project at /my/migrations/project/path Creating directory: /my/migrations/project/path Creating directory: /my/migrations/project/path/migrations Creating file: /my/migrations/project/path/ksql-migrations.properties Writing to config file: ksql.server.url=http://localhost:8088 ... Migrations project directory created successfully Execution time: 0.0080 seconds ``` -------------------------------- ### Configure ksqlDB Server with Java System Properties Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/install-ksqldb-with-docker.md This command runs a ksqlDB Server container, mounting a local volume and configuring settings like service ID and query file path using Java system properties via the KSQL_OPTS environment variable. Replace /path/on/host and /path/in/container/ with your actual paths. ```bash docker run -d \ -v /path/on/host:/path/in/container/ \ -e KSQL_BOOTSTRAP_SERVERS=localhost:9092 \ -e KSQL_OPTS="-Dksql.service.id=ksql_service_3_ -Dksql.queries.file=/path/in/container/queries.sql" \ confluentinc/ksqldb-server:{{ site.ksqldbversion }} ``` -------------------------------- ### Start Persistent Query Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-clients/java-client.md Starts a persistent query that reads from the earliest offset of a stream and aggregates data. Properties like 'auto.offset.reset' can be configured. ```java String sql = "CREATE TABLE ORDERS_BY_USER AS " + "SELECT USER_ID, COUNT(*) as COUNT " + "FROM ORDERS GROUP BY USER_ID EMIT CHANGES;"; Map properties = Collections.singletonMap("auto.offset.reset", "earliest"); ExecuteStatementResult result = client.executeStatement(sql, properties).get(); System.out.println("Query ID: " + result.queryId().orElse("")); ``` -------------------------------- ### Start Docker Compose Stack Source: https://github.com/confluentinc/ksql/blob/master/docs/how-to-guides/use-connector-management.md This command brings up the services defined in the docker-compose.yml file, including Kafka, Schema Registry, and ksqlDB server. ```bash docker-compose up ``` -------------------------------- ### ksqlDB Server Configuration via Environment Variables Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/installing.md Example of specifying ksqlDB Server configuration parameters using the environment key in a stack file. KSQL_ prefixed settings are for ksqlDB, while others can configure Kafka producer/consumer or Streams API. ```yaml environment: KSQL_LISTENERS: http://0.0.0.0:8088 KSQL_BOOTSTRAP_SERVERS: localhost:9092 ``` -------------------------------- ### Windowed ROWKEY data change example Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/upgrading.md This example demonstrates how the ROWKEY column's content and type change in windowed queries from v0.7 onwards. ```sql -- assuming `input` is a windowed stream: CREATE STREAM output AS SELECT ROWKEY as KEY_COPY, ID, NAME From input WHERE COUNT > 100; ``` -------------------------------- ### Configure CLI for Basic HTTP Authentication Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/server-config/security.md Start the ksqlDB CLI with user and password credentials for Basic HTTP authentication. ```bash bin/ksql --user fred --password letmein http://localhost:8088 ``` -------------------------------- ### Example HTTP request for ksqlDB query Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-rest-api/query-endpoint.md This is an example of an HTTP POST request to the ksqlDB query endpoint. It includes the ksql query and optional streams properties. ```http POST /query HTTP/1.1 Accept: application/vnd.ksql.v1+json Content-Type: application/vnd.ksql.v1+json { "ksql": "SELECT * FROM pageviews EMIT CHANGES;";, "streamsProperties": { "ksql.streams.auto.offset.reset": "earliest" } } ``` -------------------------------- ### ksqlDB Server Settings with Basic Auth Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/server-config/security.md Example ksqlDB server settings demonstrating Basic authentication, specifying the realm and authorized roles. ```properties authentication.method=BASIC authentication.realm=KsqlServer-Props authentication.roles=admin ``` -------------------------------- ### Example ksqlDB CLI Output Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/install-ksqldb-with-docker.md This shows the expected output when successfully connecting the ksqlDB CLI to a ksqlDB Server, including version information and the prompt for entering queries. ```text ... Copyright 2017-2020 Confluent Inc. CLI v{{ site.ksqldbversion }}, Server v{{ site.ksqldbversion }} located at http://ec2-blah.us-blah.compute.amazonaws.com:8080 Having trouble? Type 'help' (case-insensitive) for a rundown of how things work! ksql> ``` -------------------------------- ### Install Debezium MySQL Connector Source: https://github.com/confluentinc/ksql/blob/master/docs/tutorials/materialized.md Install the Debezium MySQL connector using the confluent-hub client. This command downloads the connector and places its JAR files into the specified directory. ```bash confluent-hub install --component-dir confluent-hub-components --no-prompt debezium/debezium-connector-mysql:1.1.0 ``` -------------------------------- ### Example of SHOW QUERIES Output Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-reference/show-queries.md Demonstrates the typical output format when listing running queries in ksqlDB, including Query ID, Type, Status, Sink Name, Sink Kafka Topic, and the Query String. ```sql ksql> show queries; Query ID | Query Type | Status | Sink Name | Sink Kafka Topic | Query String ------------------------------------------------------------------------------------------------------------ CSAS_TEST_0 | PERSISTENT | RUNNING:2 | TEST | TEST | CREATE STREAM TEST WITH (KAFKA_TOPIC='TEST', PARTITIONS=1, REPLICAS=1) AS SELECT *FROM KSQL_PROCESSING_LOG KSQL_PROCESSING_LOG EMIT CHANGES; ------------------------------------------------------------------------------------------------------------ For detailed information on a Query run: EXPLAIN ; ``` -------------------------------- ### Example Metastore Backup File Content Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-31-metastore-backups.md Illustrates the format of a KSQL metastore backup file, showing commands with keys and values separated by a colon. ```json "stream/`TEST1`/create":{"statement":"CREATE STREAM TEST1(ID INT) WITH(KAFKA_TOPIC='test1', VALUE_FORMAT='JSON');","streamsProperties":{}, ... } "stream/`TEST2`/create":{"statement":"CREATE STREAM TEST2(ID INT) WITH(KAFKA_TOPIC='test2', VALUE_FORMAT='JSON');","streamsProperties":{}, ... } "stream/`TEST1`/drop":{"statement":"DROP STREAM TEST1;","streamsProperties":{}, ... } ``` -------------------------------- ### Create a Directory for Connectors Source: https://github.com/confluentinc/ksql/blob/master/docs/how-to-guides/use-connector-management.md Create a directory to store downloaded connector JAR files before starting ksqlDB in embedded mode. ```bash mkdir confluent-hub-components ``` -------------------------------- ### List Properties Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/appendix.md Use LIST or SHOW to display all configured properties. ```SQL LIST PROPERTIES; ``` ```SQL SHOW PROPERTIES; ``` -------------------------------- ### ksqlDB REST API Request Example Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-rest-api/ksql-endpoint.md An example HTTP POST request to the /ksql endpoint, demonstrating the structure for sending multiple ksqlDB statements and stream properties. ```http POST /ksql HTTP/1.1 Accept: application/vnd.ksql.v1+json Content-Type: application/vnd.ksql.v1+json { "ksql": "CREATE STREAM pageviews_home AS SELECT * FROM pageviews_original WHERE pageid='home'; CREATE STREAM pageviews_alice AS SELECT * FROM pageviews_original WHERE userid='alice';", "streamsProperties": { "ksql.streams.auto.offset.reset": "earliest" } } ``` -------------------------------- ### Example ksqlDB table response with delete Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-rest-api/query-endpoint.md A sample response from the ksqlDB query endpoint when the result is a table. This example shows a row deletion, indicated by the 'row.tombstone' field. ```http HTTP/1.1 200 OK Content-Type: application/vnd.ksql.v1+json Transfer-Encoding: chunked ... {"header":{"queryId":"_confluent_id_34",schema":"`ROWTIME` BIGINT, `NAME` STRING, `ID` INT"}} {"row":{"columns":[1524760769983,"alice",10]}}, {"row":{"columns":[null,null,10],"tombstone":true}} ... ``` -------------------------------- ### Prohibited Property Response Example Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-rest-api/is_valid_property-endpoint.md This is an example of the JSON response received when a property is prohibited from setting by the ksqlDB server. It includes an error code and a message detailing the issue. ```json { "@type": "generic_error", "error_code": 40000, "message": "One or more properties overrides set locally are prohibited by the KSQL server (use UNSET to reset their default value): [ksql.service.id]" } ``` -------------------------------- ### Using WindowStart() and WindowEnd() UDAFs (Deprecated) Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/upgrading.md This example shows the deprecated usage of WindowStart() and WindowEnd() UDAFs to access window bounds in GROUP BY queries. ```sql SELECT ROWKEY, WindowStart(), WindowEnd(), COUNT() from input WINDOW SESSION (30 SECONDS) GROUP BY ROWKEY; ``` -------------------------------- ### Tier 3: Aggregate Query Examples Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-8-queryable-state-stores.md Provides examples of aggregate queries, including COUNT and SUM, which can be performed over materialized views. Supports grouping and time-based filtering. ```sql SELECT count(*) FROM table WHERE ROWKEY < 10 ``` ```sql SELECT day(hour), sum(count) FROM table GROUP BY day ``` ```sql SELECT sum(count) FROM table WHERE ts >= ‘2019-01-01’ AND ts < ‘2019-01-02’ ``` -------------------------------- ### Docker Compose for Event-Driven Microservice Stack Source: https://github.com/confluentinc/ksql/blob/master/docs/tutorials/event-driven-microservice.md Defines the services required for the tutorial: Kafka, Schema Registry, ksqlDB Server, and ksqlDB CLI. Ensure you have Docker and Docker Compose installed to run this configuration. ```yaml --- version: '2' services: kafka: image: confluentinc/cp-server:{{ site.cprelease }} hostname: kafka container_name: kafka ports: - "29092:29092" environment: KAFKA_NODE_ID: 1 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:39093 KAFKA_LISTENERS: PLAINTEXT://kafka:9092,CONTROLLER://kafka:39093,PLAINTEXT_HOST://0.0.0.0:29092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LOG_DIRS: /tmp/kraft-combined-logs CLUSTER_ID: 9-fLubcIRYqQaWkrwEThnQ schema-registry: image: confluentinc/cp-schema-registry:{{ site.cprelease }} hostname: schema-registry container_name: schema-registry depends_on: - kafka ports: - "8081:8081" environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "PLAINTEXT://kafka:9092" ksqldb-server: image: confluentinc/ksqldb-server:{{ site.ksqldbversion }} hostname: ksqldb-server container_name: ksqldb-server depends_on: - kafka - schema-registry ports: - "8088:8088" environment: KSQL_LISTENERS: "http://0.0.0.0:8088" KSQL_BOOTSTRAP_SERVERS: "kafka:9092" KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081" KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true" KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true" # Configuration to embed Kafka Connect support. KSQL_CONNECT_GROUP_ID: "ksql-connect-cluster" KSQL_CONNECT_BOOTSTRAP_SERVERS: "kafka:9092" KSQL_CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter" KSQL_CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter" KSQL_CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081" KSQL_CONNECT_CONFIG_STORAGE_TOPIC: "_ksql-connect-configs" KSQL_CONNECT_OFFSET_STORAGE_TOPIC: "_ksql-connect-offsets" KSQL_CONNECT_STATUS_STORAGE_TOPIC: "_ksql-connect-statuses" KSQL_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1 KSQL_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1 KSQL_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1 KSQL_CONNECT_PLUGIN_PATH: "/usr/share/kafka/plugins" ksqldb-cli: image: confluentinc/ksqldb-cli:{{ site.ksqldbversion }} container_name: ksqldb-cli depends_on: - kafka - ksqldb-server entrypoint: /bin/sh tty: true ``` -------------------------------- ### ksqlDB Single-Field Serialization Examples Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/serialization.md Demonstrates creating streams with default and explicit single-field value wrapping configurations. Use these examples to control how single-field values are serialized in topics. ```sql -- Assuming system configuration is at the default: -- ksql.persistence.wrap.single.values=true -- creates a stream, picking up the system default of wrapping values. -- the serialized value is expected to be wrapped. -- if the serialized forms do not match the expected wrapping it will result in a deserialization error. CREATE STREAM IMPLICIT_SOURCE (NAME STRING) WITH (...); -- override 'ksql.persistence.wrap.single.values' to false -- the serialized value is expected to not be unwrapped. CREATE STREAM EXPLICIT_SOURCE (ID INT) WITH (WRAP_SINGLE_VALUE=false, ...); -- results in an error as the value schema is multi-field CREATE STREAM BAD_SOURCE (ID INT, NAME STRING) WITH (WRAP_SINGLE_VALUE=false, ...); -- creates a stream, picking up the system default of wrapping values. -- the serialized values in the sink topic will be wrapped. CREATE STREAM IMPLICIT_SINK AS SELECT ID FROM S EMIT CHANGES; -- override 'ksql.persistence.wrap.single.values' to false -- the serialized values will not be wrapped. CREATE STREAM EXPLICIT_SINK WITH(WRAP_SINGLE_VALUE=false) AS SELECT ID FROM S EMIT CHANGES; -- results in an error as the value schema is multi-field CREATE STREAM BAD_SINK WITH(WRAP_SINGLE_VALUE=true) AS SELECT ID, COST FROM S EMIT CHANGES; ``` -------------------------------- ### Hopping Window Query Example Source: https://github.com/confluentinc/ksql/blob/master/docs/concepts/time-and-windows-in-ksqldb-queries.md Count pageviews for a specific region and gender within a hopping window. This example demonstrates a 30-second window that advances every 10 seconds. ```sql SELECT regionid, COUNT(*) FROM pageviews WINDOW HOPPING (SIZE 30 SECONDS, ADVANCE BY 10 SECONDS) WHERE UCASE(gender)='FEMALE' AND LCASE (regionid) LIKE '%_6' GROUP BY regionid EMIT CHANGES; ``` -------------------------------- ### Run ksqlDB Server and Connect ksqlDB CLI via Docker Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/installation/install-ksqldb-with-docker.md This example demonstrates running a ksqlDB Server in detached mode and then connecting to it using the ksqlDB CLI, both within Docker containers. It highlights environment variable configurations for the server. ```bash # Run ksqlDB Server. docker run -d -p 10.0.0.11:8088:8088 \ -e KSQL_BOOTSTRAP_SERVERS=localhost:9092 \ -e KSQL_OPTS="-Dksql.service.id=ksql_service_3_ -Dlisteners=http://0.0.0.0:8088/" \ confluentinc/ksqldb-server:{{ site.ksqldbversion }} # Connect the ksqlDB CLI to the server. docker run -it confluentinc/ksqldb-cli ksql http://10.0.0.11:8088 ``` -------------------------------- ### Incompatible Physical Schema Example (Type Mismatch) Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/schema-inference-with-id.md An example of an incompatible AVRO physical schema due to a type mismatch for the 'pageId' field. This schema would cause serialization errors. ```json { "schema": { "type": "record", "name": "PageViewNewSchema", "namespace": "io.confluent.ksql.avro_schemas", "fields": [ { "name": "pageId", "type": "string", "default": "id" }, { "name": "ts", "type": "int", "default": 123 } ] } } ``` -------------------------------- ### Tier 2: Generic Multi-Shard Query Examples Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-8-queryable-state-stores.md Shows examples of generic multi-shard queries that allow arbitrary WHERE clauses on any columns, enabling flexible data selection across shards. ```sql SELECT … FROM table WHERE x = ‘x’ AND y = ‘y’ ``` ```sql SELECT … FROM table WHERE str LIKE ‘%substr%’ ``` ```sql SELECT … FROM table WHERE x < y ``` -------------------------------- ### Example KSQL Query for Compatibility Testing Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-6-execution-plans.md This example query is used to illustrate compatibility breaking changes that current tests might not catch. It involves stream creation, aggregation, and filtering. ```SQL CREATE STREAM FOO (X INT, Y INT, ...) ...; CREATE TABLE AGG AS SELECT X, Y, COUNT(*) FROM FOO WHERE X=3 GROUP BY X, Y HAVING COUNT(*) > 10 AND Y > 5; ``` -------------------------------- ### Example of implicit ROWKEY leading to error Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-29-explicit-keys.md This example demonstrates the confusion arising from implicit ROWKEY creation when a stream is created without an explicit key and then used to build another stream. ```SQL ksal> CREATE STREAM S (ID INT, NAME STRING) WITH (...); Stream S Created. ksal> CREATE STREAM S2 AS SELECT ID, NAME FROM S; Key missing from projection. See https://cnfl.io/2LV7ouS. The query used to build `S2` must include the key column ROWKEY in its projection. ``` -------------------------------- ### INSTR Function Signature Source: https://github.com/confluentinc/ksql/blob/master/docs/developer-guide/ksqldb-reference/scalar-functions.md The INSTR function finds the starting position of a substring within a string, with optional parameters for starting position and occurrence count. It returns 0 if the substring is not found. ```sql INSTR(string, substring, [position], [occurrence]) ``` -------------------------------- ### Example Query Execution Plan Output Source: https://github.com/confluentinc/ksql/blob/master/docs/operate-and-deploy/capacity-planning.md This output shows the execution plan for a ksqlDB query, detailing the project, aggregate, and source stages. ```text Type : QUERY SQL : CREATE TABLE pageviews_by_page AS SELECT pageid, COUNT(*) FROM pageviews_original GROUP BY pageid EMIT CHANGES; Execution plan -------------- > [ PROJECT ] Schema: [PAGEID : STRING , KSQL_COL_1 : INT64]. > [ AGGREGATE ] Schema: [PAGEVIEWS_ORIGINAL.PAGEID : STRING , PAGEVIEWS_ORIGINAL.ROWTIME : INT64 , KSQL_AGG_VARIABLE_0 : INT64]. > [ PROJECT ] Schema: [PAGEVIEWS_ORIGINAL.PAGEID : STRING , PAGEVIEWS_ORIGINAL.ROWTIME : INT64]. > [ SOURCE ] Schema: [PAGEVIEWS_ORIGINAL.ROWTIME : INT64 , PAGEVIEWS_ORIGINAL.VIEWTIME : INT64 , PAGEVIEWS_ORIGINAL.USERID : STRING , PAGEVIEWS_ORIGINAL.PAGEID : STRING]. Processing topology ------------------- Topologies: Sub-topology: 0 Source: KSTREAM-SOURCE-0000000000 (topics: [pageviews]) --> KSTREAM-MAPVALUES-0000000001 Processor: KSTREAM-MAPVALUES-0000000001 (stores: []) --> KSTREAM-TRANSFORMVALUES-0000000002 <-- KSTREAM-SOURCE-0000000000 Processor: KSTREAM-TRANSFORMVALUES-0000000002 (stores: []) --> KSTREAM-MAPVALUES-0000000003 <-- KSTREAM-MAPVALUES-0000000001 Processor: KSTREAM-MAPVALUES-0000000003 (stores: []) --> KSTREAM-FILTER-0000000004 <-- KSTREAM-TRANSFORMVALUES-0000000002 Processor: KSTREAM-FILTER-0000000004 (stores: []) --> Aggregate-groupby <-- KSTREAM-MAPVALUES-0000000003 Processor: Aggregate-groupby (stores: []) --> KSTREAM-FILTER-0000000009 <-- KSTREAM-FILTER-0000000004 Processor: KSTREAM-FILTER-0000000009 (stores: []) --> KSTREAM-SINK-0000000008 <-- Aggregate-groupby Sink: KSTREAM-SINK-0000000008 (topic: KSTREAM-AGGREGATE-STATE-STORE-0000000006-repartition) <-- KSTREAM-FILTER-0000000009 Sub-topology: 1 Source: KSTREAM-SOURCE-0000000010 (topics: [KSTREAM-AGGREGATE-STATE-STORE-0000000006-repartition]) --> KSTREAM-AGGREGATE-0000000007 Processor: KSTREAM-AGGREGATE-0000000007 (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000006]) --> KTABLE-MAPVALUES-0000000011 <-- KSTREAM-SOURCE-0000000010 Processor: KTABLE-MAPVALUES-0000000011 (stores: []) --> KTABLE-TOSTREAM-0000000012 <-- KSTREAM-AGGREGATE-0000000007 Processor: KTABLE-TOSTREAM-0000000012 (stores: []) --> KSTREAM-MAPVALUES-0000000013 <-- KTABLE-TOSTREAM-0000000012 Processor: KSTREAM-MAPVALUES-0000000013 (stores: []) --> KSTREAM-SINK-0000000014 <-- KTABLE-TOSTREAM-0000000012 Sink: KSTREAM-SINK-0000000014 (topic: PAGEVIEWS_BY_PAGE) <-- KSTREAM-MAPVALUES-0000000013 ``` -------------------------------- ### Valid ksqlDB SQL Input Examples Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/sql/syntax/lexical-structure.md These examples demonstrate syntactically valid ksqlDB SQL input, including INSERT, CREATE STREAM, and SELECT statements. Ensure statements are terminated with a semicolon. ```sql INSERT INTO s1 (a, b) VALUES ('k1', 'v1'); ``` ```sql CREATE STREAM s2 AS SELECT a, b FROM s1 EMIT CHANGES; ``` ```sql SELECT * FROM t1 WHERE k1='foo' EMIT CHANGES; ``` -------------------------------- ### SQL Script for Creating Stream and View Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-32-sql-testing-tool.md Example SQL script to create a stream 'foo' and a view 'bar' based on 'foo'. ```sql -- 1_create_foo_bar.sql CREATE STREAM foo (id INT KEY, col1 VARCHAR) WITH (...) CREATE STREAM bar AS SELECT * from FOO; ``` -------------------------------- ### Example of Hive's explode table function Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-9-table-functions.md This example demonstrates how Apache Hive's 'explode' function can transform an array into multiple rows. It is analogous to ksqlDB's table functions. ```sql select explode(array('A','B','C')); ``` -------------------------------- ### KSQL Migrations Properties Configuration Source: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-42-schema-migrations-tool.md Example configuration file for setting up ksqlDB server connection details and migration-specific parameters. Customize these properties for your environment. ```properties # Server URL and authentication ksql.server.url=http://localhost:8080 ksql.username=user1 ksql.password=pass1 # TLS configs, including for mTLS ssl.truststore.location=/path/to/truststore ssl.truststore.password=pass ssl.keystore.location=/path/to/keystore ssl.keystore.password=pass ssl.key.password=pass # Migrations details ksql.migrations.stream.name=migration_events ksql.migrations.table.name=schema_version ksql.migrations.stream.topic.name=default_ksql_migration_events ksql.migrations.table.topic.name=default_ksql_schema_version ksql.migrations.topic.replicas=1 ``` -------------------------------- ### Example ksqlDB Execution Plan Loggers Source: https://github.com/confluentinc/ksql/blob/master/docs/reference/processing-log.md This example shows the hierarchical logger names within a ksqlDB execution plan. These names help trace processing events back to specific query steps. ```plaintext Execution plan -------------- > [ SINK ] | Schema: [VIEWTIME : BIGINT, KSQL_COL_1 : VARCHAR, KSQL_COL_2 : VARCHAR] | Logger: processing.CSAS_PAGEVIEWS_UPPER_0.PAGEVIEWS_UPPER > [ PROJECT ] | Schema: [VIEWTIME : BIGINT, KSQL_COL_1 : VARCHAR, KSQL_COL_2 : VARCHAR] | Logger: processing.CSAS_PAGEVIEWS_UPPER_0.Project > [ SOURCE ] | Schema: [PAGEVIEWS_ORIGINAL.ROWTIME : BIGINT, PAGEVIEWS_ORIGINAL.VIEWTIME : BIGINT, PAGEVIEWS_ORIGINAL.USERID : VARCHAR, PAGEVIEWS_ORIGINAL.PAGEID : VARCHAR] | Logger: processing.CSAS_PAGEVIEWS_UPPER_0.KsqlTopic ```