### Publish example metrics to Kafka Source: https://druid.apache.org/docs/latest/development/extensions-core/protobuf Generates example metrics and pipes them to the Protobuf publisher script, which requires the `protobuf` and `kafka-python` modules. Run from the Druid installation directory. ```bash ./bin/generate-example-metrics | python /quickstart/protobuf/pb_publisher.py ``` -------------------------------- ### Regular expression filter example Source: https://druid.apache.org/docs/latest/querying/filters An example of a regex filter that matches values starting with '50.' in a specified column. ```json { "type": "regex", "dimension": "someColumn", "pattern": ^50.* } ``` -------------------------------- ### Example with JDK 17+ parameters Source: https://druid.apache.org/docs/latest/operations/dump-segment Full example of running the tool with the JDK 17 required opens, classpath, and argument list. ```shell java --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED \ -classpath "/my/druid/lib/*" \ -Ddruid.extensions.loadList="[]" org.apache.druid.cli.Main \ tools dump-segment \ --directory /home/druid/path/to/segment/ \ --out /home/druid/output.txt ``` -------------------------------- ### Start Query Server Source: https://druid.apache.org/docs/latest/tutorials/cluster Run this command from the distribution root on your Query servers to start the service. ```bash bin/start-cluster-query-server ``` -------------------------------- ### Get Role Details Output Source: https://druid.apache.org/docs/latest/development/extensions-core/druid-basic-security Example output for the GET role details API, showing the default format including the resourceNamePattern field. ```json { "name": "druidRole2", "permissions": [ { "resourceAction": { "resource": { "name": "E", "type": "DATASOURCE" }, "action": "WRITE" }, "resourceNamePattern": "E" } ] } ``` -------------------------------- ### Get Role Details with ?simplifyPermissions Output Source: https://druid.apache.org/docs/latest/development/extensions-core/druid-basic-security Example output for the GET role details API when the ?simplifyPermissions flag is used, excluding the resourceNamePattern field. ```json { "name": "druidRole2", "users": null, "permissions": [ { "resource": { "name": "E", "type": "DATASOURCE" }, "action": "WRITE" } ] } ``` -------------------------------- ### View initial data in append_tutorial Source: https://druid.apache.org/docs/latest/tutorials/tutorial-append-data Run this query in a new tab to view the contents of the 'append_tutorial' datasource after loading the sample data. ```sql SELECT * FROM "append_tutorial" ``` -------------------------------- ### Start Data Server Source: https://druid.apache.org/docs/latest/tutorials/cluster Run this command from the distribution root on your Data servers to start the service. ```bash bin/start-cluster-data-server ``` -------------------------------- ### Cardinality example: distinct starting characters of last names Source: https://druid.apache.org/docs/latest/querying/hll-old Example cardinality aggregator using an extraction function to count distinct first characters of last names. ```json { "type": "cardinality", "name": "distinct_last_name_first_char", "fields": [ { "type" : "extraction", "dimension" : "last_name", "outputName" : "last_name_first_char", "extractionFn" : { "type" : "substring", "index" : 0, "length" : 1 } } ], "byRow" : true } ``` -------------------------------- ### JDBC Driver: Full example Source: https://druid.apache.org/docs/latest/querying/query-context A complete Java example demonstrating how to connect to Druid via JDBC and execute a query with context properties. ```java import java.sql.*; import java.util.Properties; public class JdbcDruid { public static void main(String args[]) { // Connect to /druid/v2/sql/avatica/ on your Broker. String url = "jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica/;transparent_reconnection=true"; // The query you want to run. String query = "SELECT * FROM wikipedia WHERE user = 'BlueMoon2662'"; // Set any connection context parameters you need here. Properties connectionProperties = new Properties(); connectionProperties.setProperty("sqlTimeZone", "America/Los_Angeles"); try (Connection connection = DriverManager.getConnection(url, connectionProperties)) { try ( final Statement statement = connection.createStatement(); final ResultSet rs = statement.executeQuery(query) ) { while (rs.next()) { // process result set Timestamp timeStamp = rs.getTimestamp("__time"); System.out.println(timeStamp); } } } catch (Exception e) { System.out.println(e.toString()); } } } ``` -------------------------------- ### Get task log response Source: https://druid.apache.org/docs/latest/api-reference/tasks-api Example output of a task log containing event details. ```text 2023-07-03T22:11:17,891 INFO [qtp1251996697-122] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Sequence[index_kafka_social_media_0e905aa31037879_0] end offsets updated from [{0=9223372036854775807}] to [{0=230985}]. 2023-07-03T22:11:17,900 INFO [qtp1251996697-122] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Saved sequence metadata to disk: [SequenceMetadata{sequenceId=0, sequenceName='index_kafka_social_media_0e905aa31037879_0', assignments=[0], startOffsets={0=230985}, exclusiveStartPartitions=[], endOffsets={0=230985}, sentinel=false, checkpointed=true}] 2023-07-03T22:11:17,901 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Received resume command, resuming ingestion. 2023-07-03T22:11:17,901 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Finished reading partition[0], up to[230985]. 2023-07-03T22:11:17,902 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-kafka-supervisor-dcanhmig-1, groupId=kafka-supervisor-dcanhmig] Resetting generation and member id due to: consumer pro-actively leaving the group ``` -------------------------------- ### Create mvd-example datasource Source: https://druid.apache.org/docs/latest/querying/sql-functions Use this SQL query to create the 'mvd-example' datasource, which demonstrates the use of multi-value string dimensions. ```sql REPLACE INTO "mvd-example" OVERWRITE ALL WITH "ext" AS ( SELECT * FROM TABLE( EXTERN( '{"type":"inline","data":"{\"timestamp\": \"2011-01-12T00:00:00.000Z\", \"label\": \"row1\", \"tags\": [\"t1\",\"t2\",\"t3\"]}\n{\"timestamp\": \"2011-01-13T00:00:00.000Z\", \"label\": \"row2\", \"tags\": [\"t3\",\"t4\",\"t5\"]}\n{\"timestamp\": \"2011-01-14T00:00:00.000Z\", \"label\": \"row3\", \"tags\": [\"t5\",\"t6\",\"t7\"]}\n{\"timestamp\": \"2011-01-14T00:00:00.000Z\", \"label\": \"row4\", \"tags\": []}"}', '{"type":"json"}', '[{"name":"timestamp", "type":"STRING"},{"name":"label", "type":"STRING"},{"name":"tags", "type":"ARRAY"}]' ) ) ) SELECT TIME_PARSE("timestamp") AS "__time", "label", ARRAY_TO_MV("tags") AS "tags" FROM "ext" PARTITIONED BY DAY ``` -------------------------------- ### POSITION example: find substring index in OriginCityName Source: https://druid.apache.org/docs/latest/querying/sql-functions Returns the one-based index of substring 'PR' in the OriginCityName column starting from index 5. If not found, returns 0. Example uses the flight-carriers datasource. ```sql SELECT "OriginCityName" AS "origin_city", POSITION('PR' IN "OriginCityName" FROM 5) AS "index" FROM "flight-carriers" LIMIT 2 ``` -------------------------------- ### Runtime property example Source: https://druid.apache.org/docs/latest/querying/query-context An example of setting the debug context parameter globally. ```text druid.query.default.context.debug=true ``` -------------------------------- ### Get task status response Source: https://druid.apache.org/docs/latest/api-reference/tasks-api Example JSON response containing task metadata and status information. ```json { "task": "query-223549f8-b993-4483-b028-1b0d54713cad", "status": { "id": "query-223549f8-b993-4483-b028-1b0d54713cad", "groupId": "query-223549f8-b993-4483-b028-1b0d54713cad", "type": "query_controller", "createdTime": "2023-06-22T22:11:28.367Z", "queueInsertionTime": "1970-01-01T00:00:00.000Z", "statusCode": "RUNNING", "status": "RUNNING", "runnerStatusCode": "RUNNING", "duration": -1, "location": {"host": "localhost", "port": 8100, "tlsPort": -1}, "dataSource": "wikipedia_api", "errorMsg": null } } ``` -------------------------------- ### Quick Start: Load the extension and minimal Consul configuration Source: https://druid.apache.org/docs/latest/development/extensions-contrib/consul Loads `druid-consul-extensions` in the extensions load list and sets minimal Consul connection settings (host and port), the service prefix, and the required HTTP-based configurations for server view and indexer runner. Also enables Consul-based leader election for Coordinator and Overlord. This is for development/quick start; production requires additional security and TLS options in the Configuration section. ```properties # Load the extension druid.extensions.loadList=["druid-consul-extensions"] # Minimal Consul configuration druid.discovery.type=consul druid.discovery.consul.connection.host=localhost druid.discovery.consul.connection.port=8500 druid.discovery.consul.service.servicePrefix=druid # Required HTTP-based configurations druid.serverview.type=http druid.indexer.runner.type=httpRemote # Enable Consul-based leader election druid.coordinator.selector.type=consul druid.indexer.selector.type=consul ``` -------------------------------- ### SQL Join Example Source: https://druid.apache.org/docs/latest/querying/datasource Joins 'sales' with 'countries' using 'store' as the join key to get sales by country. ```SQL -- Joins "sales" with "countries" (using "store" as the join key) to get sales by country. SELECT store_to_country.v AS country, SUM(sales.revenue) AS country_revenue FROM sales INNER JOIN lookup.store_to_country ON sales.store = store_to_country.k GROUP BY countries.v ``` -------------------------------- ### Druid startup output Source: https://druid.apache.org/docs/latest/tutorials Example output displayed in the terminal when the start-druid script successfully initializes the cluster services. ```text $ ./bin/start-druid [Tue Nov 29 16:31:06 2022] Starting Apache Druid. [Tue Nov 29 16:31:06 2022] Open http://localhost:8888/ in your browser to access the web console. [Tue Nov 29 16:31:06 2022] Or, if you have enabled TLS, use https on port 9088. [Tue Nov 29 16:31:06 2022] Starting services with log directory [/apache-druid-37.0.0/log]. [Tue Nov 29 16:31:06 2022] Running command[zk]: bin/run-zk conf [Tue Nov 29 16:31:06 2022] Running command[broker]: bin/run-druid broker /apache-druid-37.0.0/conf/druid/single-server/quickstart '-Xms1187m -Xmx1187m -XX:MaxDirectMemorySize=791m' [Tue Nov 29 16:31:06 2022] Running command[router]: bin/run-druid router /apache-druid-37.0.0/conf/druid/single-server/quickstart '-Xms128m -Xmx128m' [Tue Nov 29 16:31:06 2022] Running command[coordinator-overlord]: bin/run-druid coordinator-overlord /apache-druid-37.0.0/conf/druid/single-server/quickstart '-Xms1290m -Xmx1290m' [Tue Nov 29 16:31:06 2022] Running command[historical]: bin/run-druid historical /apache-druid-37.0.0/conf/druid/single-server/quickstart '-Xms1376m -Xmx1376m -XX:MaxDirectMemorySize=2064m' [Tue Nov 29 16:31:06 2022] Running command[middleManager]: bin/run-druid middleManager /apache-druid-37.0.0/conf/druid/single-server/quickstart '-Xms64m -Xmx64m' '-Ddruid.worker.capacity=2 -Ddruid.indexer.runner.javaOptsArray=["-server","-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-XX:+ExitOnOutOfMemoryError","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager","-Xms256m","-Xmx256m","-XX:MaxDirectMemorySize=256m"]' ``` -------------------------------- ### Get active supervisor objects response Source: https://druid.apache.org/docs/latest/api-reference/supervisor-api Example JSON response containing detailed supervisor object information. ```json [ { "id": "wikipedia_stream", "state": "RUNNING", "detailedState": "CONNECTING_TO_STREAM", "healthy": true, "spec": { "type": "kafka", "spec": { "dataSchema": { "dataSource": "wikipedia_stream", "timestampSpec": { "column": "__time", "format": "iso", "missingValue": null }, "dimensionsSpec": { "dimensions": [ { "type": "string", "name": "username", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "string", "name": "post_title", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": true }, { "type": "long", "name": "views", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": false }, { "type": "long", "name": "upvotes", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": false }, { "type": "long", "name": "comments", "multiValueHandling": "SORTED_ARRAY", "createBitmapIndex": false }, { "type": "string", ``` -------------------------------- ### Get active supervisor IDs response Source: https://druid.apache.org/docs/latest/api-reference/supervisor-api Example JSON response containing a list of active supervisor IDs. ```json [ "wikipedia_stream", "social_media" ] ``` -------------------------------- ### Insert sample data into latest_by_tutorial2 Source: https://druid.apache.org/docs/latest/tutorials/tutorial-latest-by Use this REPLACE INTO query to ingest sample user point data into the latest_by_tutorial2 datasource for testing the LATEST_BY function. ```sql REPLACE INTO "latest_by_tutorial2" OVERWRITE ALL WITH "ext" AS ( SELECT * FROM TABLE( EXTERN( '{"type":"inline","data":"{\"timestamp\":\"2024-01-01T01:00:00Z\",\"updated_timestamp\":\"2024-01-01T01:00:00Z\",\"user_id\":\"funny_bunny1\", \"points\":10}\n{\"timestamp\":\"2024-01-01T01:05:00Z\",\"updated_timestamp\":\"2024-01-01T01:05:00Z\",\"user_id\":\"funny_bunny1\", \"points\":30}\n{\"timestamp\": \"2024-01-01T02:00:00Z\",\"updated_timestamp\":\"2024-01-01T02:00:00Z\",\"user_id\":\"funny_bunny1\", \"points\":35}\n{\"timestamp\":\"2024-01-01T02:00:00Z\",\"updated_timestamp\":\"2024-01-01T02:00:00Z\",\"user_id\":\"silly_monkey2\", \"points\":30}\n{\"timestamp\":\"2024-01-01T02:00:00Z\",\"updated_timestamp\":\"2024-01-01T02:05:00Z\",\"user_id\":\"silly_monkey2\", \"points\":55}\n{\"timestamp\":\"2024-01-01T03:00:00Z\",\"updated_timestamp\":\"2024-01-01T03:00:00Z\",\"user_id\":\"funny_bunny1\", \"points\":40}"}', '{"type":"json"}' ) ) EXTEND ("timestamp" VARCHAR, "updated_timestamp" VARCHAR, "user_id" VARCHAR, "points" BIGINT) ) SELECT TIME_PARSE("timestamp") AS "__time", "updated_timestamp", "user_id", "points" FROM "ext" PARTITIONED BY DAY ``` -------------------------------- ### Retrieve Query Results via HTTP Source: https://druid.apache.org/docs/latest/api-reference/sql-api Example HTTP GET request to retrieve the results of a specific query. ```http GET /druid/v2/sql/statements/query-f3bca219-173d-44d4-bdc7-5002e910352f/results HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` -------------------------------- ### Basic Authorizer Example Configuration Source: https://druid.apache.org/docs/latest/development/extensions-core/druid-basic-security An example showing how to register a named authorizer and set its type to basic. ```properties # Authorizer druid.auth.authorizers=["MyBasicMetadataAuthorizer"] druid.auth.authorizer.MyBasicMetadataAuthorizer.type=basic ``` -------------------------------- ### Start the Druid cluster with Docker Compose Source: https://druid.apache.org/docs/latest/tutorials/docker Run this command from the directory containing the configuration files (the directory you created or distribution/docker/ in a local Druid installation). Use docker compose up to attach a shell, or docker compose up -d to run in the background. The web console is available at http://localhost:8888 after startup. ```bash docker compose up ``` ```bash docker compose up -d ``` -------------------------------- ### Retrieve tasks with HTTP Source: https://druid.apache.org/docs/latest/api-reference/tasks-api Example of an HTTP GET request to fetch a filtered list of tasks from the Druid cluster. ```http GET /druid/indexer/v1/tasks/?state=complete&datasource=wikipedia_api&createdTimeInterval=2015-09-12_2015-09-13&max=10&type=query_worker HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` -------------------------------- ### Install aliyun-oss-extensions with pull-deps Source: https://druid.apache.org/docs/latest/development/extensions-contrib/aliyun-oss Use this command to install the aliyun-oss-extensions extension via the pull-deps tool. Replace {YOUR_DRUID_DIR} and {YOUR_DRUID_VERSION} with your actual Druid directory and version. Run this on middle manager and historical nodes. ```bash java -classpath "{YOUR_DRUID_DIR}/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:aliyun-oss-extensions:{YOUR_DRUID_VERSION} ``` -------------------------------- ### Get Role Users Output Source: https://druid.apache.org/docs/latest/development/extensions-core/druid-basic-security Example output when using the ?full flag to retrieve the list of users associated with a role. ```json {"users":["druid"]} ``` -------------------------------- ### Get all datasource compaction configurations request Source: https://druid.apache.org/docs/latest/api-reference/automatic-compaction-api Use these examples to retrieve all datasource compaction configurations via the /druid/indexer/v1/compaction/config/datasources endpoint. ```bash curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/compaction/config/datasources" ``` ```http GET /druid/indexer/v1/compaction/config/datasources HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` -------------------------------- ### Sample data CSV Source: https://druid.apache.org/docs/latest/tutorials/tutorial-sketches-theta Sample dataset with columns date, uid, show, and episode. Used for the tutorial. ```csv date,uid,show,episode 2022-05-19,alice,Game of Thrones,S1E1 2022-05-19,alice,Game of Thrones,S1E2 2022-05-19,alice,Game of Thrones,S1E1 2022-05-19,bob,Bridgerton,S1E1 2022-05-20,alice,Game of Thrones,S1E1 2022-05-20,carol,Bridgerton,S1E2 2022-05-20,dan,Bridgerton,S1E1 2022-05-21,alice,Game of Thrones,S1E1 2022-05-21,carol,Bridgerton,S1E1 2022-05-21,erin,Game of Thrones,S1E1 2022-05-21,alice,Bridgerton,S1E1 2022-05-22,bob,Game of Thrones,S1E1 2022-05-22,bob,Bridgerton,S1E1 2022-05-22,carol,Bridgerton,S1E2 2022-05-22,bob,Bridgerton,S1E1 2022-05-22,erin,Game of Thrones,S1E1 2022-05-22,erin,Bridgerton,S1E2 2022-05-23,erin,Game of Thrones,S1E1 2022-05-23,alice,Game of Thrones,S1E1 ``` -------------------------------- ### Retrieve task report via cURL Source: https://druid.apache.org/docs/latest/api-reference/sql-ingestion-api Example of using cURL to perform a GET request for a specific task report. ```bash curl --location --request GET 'http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/task/query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e/reports' ``` -------------------------------- ### URI Lookup Configuration with Prefix and Regex Source: https://druid.apache.org/docs/latest/querying/lookups-cached-global Example of a URI lookup configuration using a prefix and a regex to match files, including heap size constraints. ```json { "type":"uri", "uriPrefix": "s3://bucket/some/key/prefix/", "fileRegex":"renames-[0-9]*\\.gz", "namespaceParseSpec":{ "format":"csv", "columns":[ "[\"key\"", "\"value\"]" ] }, "pollPeriod":"PT5M", "maxHeapPercentage": 10 } ``` -------------------------------- ### Get worker nodes response Source: https://druid.apache.org/docs/latest/api-reference/dynamic-configuration-api Example JSON response containing worker node metadata, capacity, and task information. ```json [ { "worker": { "scheme": "http", "host": "localhost:8091", "ip": "198.51.100.0", "capacity": 2, "version": "0", "category": "_default_worker_category" }, "currCapacityUsed": 0, "currParallelIndexCapacityUsed": 0, "availabilityGroups": [], "runningTasks": [], "lastCompletedTaskTime": "2023-09-29T19:13:05.505Z", "blacklistedUntil": null } ] ``` -------------------------------- ### Create array-example datasource Source: https://druid.apache.org/docs/latest/querying/sql-functions Use this SQL query to create the 'array-example' datasource using inline JSON data, which is required for testing array functions. ```sql REPLACE INTO "array-example" OVERWRITE ALL WITH "ext" AS ( SELECT * FROM TABLE( EXTERN( '{"type":"inline","data":"{\"timestamp\": \"2023-01-01T00:00:00\", \"label\": \"row1\", \"arrayString\": [\"a\", \"b\"], \"arrayLong\":[1, null,3], \"arrayDouble\":[1.1, 2.2, null]}\n{\"timestamp\": \"2023-01-01T00:00:00\", \"label\": \"row2\", \"arrayString\": [null, \"b\"], \"arrayLong\":null, \"arrayDouble\":[999, null, 5.5]}\n{\"timestamp\": \"2023-01-01T00:00:00\", \"label\": \"row3\", \"arrayString\": [], \"arrayLong\":[1, 2, 3], \"arrayDouble\":[null, 2.2, 1.1]} \n{\"timestamp\": \"2023-01-01T00:00:00\", \"label\": \"row4\", \"arrayString\": [\"a\", \"b\"], \"arrayLong\":[1, 2, 3], \"arrayDouble\":[]}\n{\"timestamp\": \"2023-01-01T00:00:00\", \"label\": \"row5\", \"arrayString\": null, \"arrayLong\":[], \"arrayDouble\":null}"}', '{"type":"json"}' ) ) EXTEND ( "timestamp" VARCHAR, "label" VARCHAR, "arrayString" VARCHAR ARRAY, "arrayLong" BIGINT ARRAY, "arrayDouble" DOUBLE ARRAY ) ) SELECT TIME_PARSE("timestamp") AS "__time", "label", "arrayString", "arrayLong", "arrayDouble" FROM "ext" PARTITIONED BY DAY ``` -------------------------------- ### Get broker dynamic configuration response Source: https://druid.apache.org/docs/latest/api-reference/dynamic-configuration-api Example JSON response containing the current Broker dynamic configuration properties. ```json { "queryBlocklist": [ { "ruleName": "block-expensive-scans", "dataSources": ["large_table"], "queryTypes": ["scan"] } ], "queryContext": { "priority": 0, "timeout": 300000 }, "perSegmentTimeoutConfig": { "large_table": { "perSegmentTimeoutMs": 10000, "monitorOnly": false } } } ```