### Start Docker Compose Services Source: https://docs.lenses.io/latest/k2k/getting-started/run-a-quick-example Command to start the services defined in the Docker Compose file in detached mode. ```bash docker compose up -d ``` -------------------------------- ### Start K2K Replicator Source: https://docs.lenses.io/latest/k2k/~gitbook/pdf Command to start the K2K replicator with a specified pipeline configuration file. This is the primary command for initiating data replication. ```bash k2k start -f my-pipeline.yaml ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-studio/accessing-headers Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-studio Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Produce Messages to Source Kafka Source: https://docs.lenses.io/latest/k2k/~gitbook/pdf Example messages to be produced to the source Kafka cluster. ```bash key1,hello key1,this key1,is key1,k2k ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/filtering-and-joins/using-multiple-topics Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-studio/querying-data Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Start Docker Compose Services Source: https://docs.lenses.io/latest/k2k/getting-started Command to start the services defined in the Docker Compose file in detached mode. ```bash docker compose up -d ``` -------------------------------- ### K2K Pipeline Configuration Example Source: https://docs.lenses.io/latest/k2k/~gitbook/pdf A sample YAML configuration file for K2K, defining source and target Kafka connections, topic replication details, and coordination settings. This file dictates how data flows between Kafka clusters. ```yaml name: "my-topic-k2k" source: kafka: connection: servers: "sourceKafka:9092" consumer: "group.id": "k2k.my-topic-k2k.source" target: kafka: connection: servers: "targetKafka:9092" replication: - source: name: source topic: "my-topic" - sink: name: sink-source-topic topic: source partition: source coordination: kafka: commit: group: "k2k.my-topic-k2k.coordination" ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/startswith Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/views-and-synonyms Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/filtering-and-joins Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/data-formats Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/limit-and-sampling Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/managing-queries Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/best-practices Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/joins Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/creating-and-deleting-kafka-topics Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/data-formats/rekeying-data Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/filtering-and-joins/enriching-data-streams Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/viewing-topic-partitions Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/filtering-and-joins/filtering-data Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/aggregations Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/supported-data-formats/protobuf Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/controlling-event-time Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-studio/deleting-data-from-compacted-topics Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/getting-started/quickstart/hands-on-walk-through-of-community-edition Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install K2K Docker Image Source: https://docs.lenses.io/latest/k2k/~gitbook/pdf Pulls the latest Docker image for the Lenses Kafka to Kafka replicator. This is the first step in setting up K2K using Docker. ```bash docker pull lensting/k2k:0.0.2-alpha ``` -------------------------------- ### Produce Messages to Source Kafka Source: https://docs.lenses.io/latest/k2k/getting-started/run-a-quick-example Example messages to be produced to the source Kafka cluster's 'my-topic' using the console producer. ```text key1,hello key1,this key1,is key1,k2k ``` -------------------------------- ### Start Docker Compose Services Source: https://docs.lenses.io/latest/k2k/~gitbook/pdf Command to start the Docker Compose services in detached mode. ```bash docker compose up -d ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/complex-types Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/concepts Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/takeright Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Full Provisioning Example Source: https://docs.lenses.io/latest/deployment/installation/linux/agent This comprehensive YAML provisioning file demonstrates connections to Kafka Cluster, Schema Registry, Kafka Connect, and Lenses HQ. It includes configurations for server addresses, ports, and security settings for each component. ```yaml lensesHq: - configuration: agentKey: value: agent_key_* port: value: 10000 server: value: current-lenses-hq.panoptes.svc.cluster.local sslEnabled: value: false name: lenses-hq tags: - hq version: 1 kafka: - configuration: kafkaBootstrapServers: value: - PLAINTEXT://prod-1-kafka-bootstrap.kafka-prod.svc.cluster.local:9092 metricsPort: value: 9999 metricsType: value: JMX protocol: value: PLAINTEXT name: kafka tags: - prod - prod-1 - us version: 1 confluentSchemaRegistry: - configuration: schemaRegistryUrls: value: - http://schema-registry-prod.kafka-prod.svc.cluster.local:8081 name: schema-registry tags: - prod - global version: 1 connect: - configuration: workers: value: - http://prod-1-connect-connect.kafka-prod.svc.cluster.local:8083 name: datalake-connect tags: - prod - us version: 1 ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/backup-and-restore Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/finding-topics-and-fields Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/aggregations/aggregating-data-in-a-table Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Provisioning Example with Multiple Connections Source: https://docs.lenses.io/latest/deployment/installation/docker/agent This YAML file provides a comprehensive provisioning example, including connections to Kafka Cluster, Schema Registry, Kafka Connect, and Lenses HQ. ```yaml lensesHq: - configuration: agentKey: value: agent_key_* port: value: 10000 server: value: current-lenses-hq.panoptes.svc.cluster.local sslEnabled: value: false name: lenses-hq tags: - hq version: 1 kafka: - configuration: kafkaBootstrapServers: value: - PLAINTEXT://prod-1-kafka-bootstrap.kafka-prod.svc.cluster.local:9092 metricsPort: value: 9999 metricsType: value: JMX protocol: value: PLAINTEXT name: kafka tags: - prod - prod-1 - us version: 1 confluentSchemaRegistry: - configuration: schemaRegistryUrls: value: - http://schema-registry-prod.kafka-prod.svc.cluster.local:8081 name: schema-registry tags: - prod - global version: 1 connect: - configuration: workers: value: - http://prod-1-connect-connect.kafka-prod.svc.cluster.local:8083 name: datalake-connect tags: - prod - us version: 1 ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Kafka Connect Role Permission Example Source: https://docs.lenses.io/latest/user-guide/iam/iam-reference This snippet provides an example of a role permission configuration for Kafka Connect operations. It specifies allowed actions and resources for managing connectors and clusters, including listing, getting details, starting, and stopping connectors. ```YAML name: global-connector-operator policy: - action: - iam:List* - iam:Get* resource: iam:* effect: allow - action: - environments:Get* - environments:List* - environments:AccessEnvironment resource: environments:* effect: allow - action: - kafka-connect:List* -kafka-connect:GetClusterDetails -kafka-connect:GetConnectorDetails -kafka-connect:StartConnector -kafka-connect:StopConnector resource: - kafka-connect:cluster:*/* - kafka-connect:connector:*/*/* effect: allow ``` -------------------------------- ### Produce Messages to Source Kafka Source: https://docs.lenses.io/latest/k2k/getting-started Example messages to be produced to the source Kafka cluster's 'my-topic' using the console producer. ```text key1,hello key1,this key1,is key1,k2k ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/global-topic-catalogue Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/complex-types/working-with-arrays Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### K2K Pipeline Configuration Source: https://docs.lenses.io/latest/k2k/getting-started/run-a-quick-example This YAML configuration file defines a K2K pipeline for replicating a Kafka topic. It specifies the source and target Kafka cluster connections, the topic to be replicated, and coordination settings. ```yaml name: "my-topic-k2k" source: kafka: connection: servers: "sourceKafka:9092" consumer: "group.id": "k2k.my-topic-k2k.source" target: kafka: connection: servers: "targetKafka:9092" replication: - source: name: source topic: "my-topic" - sink: name: sink-source-topic topic: source partition: source coordination: kafka: commit: group: "k2k.my-topic-k2k.coordination" ``` -------------------------------- ### Configure Kafka SASL/SSL and Schema Registry Source: https://docs.lenses.io/latest/k2k/~gitbook/pdf Example YAML snippets demonstrating how to configure Kafka SASL/SSL and Schema Registry properties for K2K replicator. This includes settings for consumers, producers, and the Schema Registry URL, username, and password. ```yaml pipeline.yaml with consumer and producer props for SASL and SSL # Example for Schema Registry configuration source: kafka: consumer: schema.registry.url: "http://localhost:8081" schema.registry.username: "user" schema.registry.password: "password" target: kafka: producer: schema.registry.url: "http://localhost:8081" schema.registry.username: "user" schema.registry.password: "password" coordination: kafka: consumer: schema.registry.url: "http://localhost:8081" schema.registry.username: "user" schema.registry.password: "password" ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/managing-topic-configurations Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/aggregations/time-window-aggregations Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/takeleft Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Deploy Lenses Agent on Linux Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/startswith Instructions for deploying the Lenses Agent component on a Linux system. This covers downloading and running the agent. ```Bash wget https://packages.lenses.io/lenses-agent.tar.gz tar -xzf lenses-agent.tar.gz cd lenses-agent ./bin/lenses-agent start ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/topic-settings Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/arrays Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Get Provisioning State (JavaScript) Source: https://docs.lenses.io/latest/resources/api-reference/lenses-api-spec/provisioning-state Example of fetching the provisioning state using JavaScript. This code demonstrates making a GET request to the Lenses.io API endpoint. ```javascript fetch('/api/v1/environments/{name}/proxy/api/v1/state', { method: 'GET', headers: { 'Accept': '*/*' } }) .then(response => response.json()) .then(data => console.log(data)) .catch(error => console.error('Error:', error)); ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/filtering Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/adding-metadata-and-tags-to-topics Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topology Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Create and Insert Data into Tutorial Table Source: https://docs.lenses.io/latest/user-guide/sql-studio/creating-and-deleting-kafka-topics This snippet demonstrates how to create a SQL table named 'tutorial' with specified columns and format, and then insert sample data into it. It uses SQL syntax for table creation and data insertion. ```SQL CREATE TABLE tutorial( _key STRING , name STRING , difficulty INT ) FORMAT (Avro, Avro); INSERT INTO tutorial(_key, name, difficulty) VALUES ("1", "Learn Lenses SQL", 3), ("2", "Learn Quantum Physics", 10), ("3", "Learn French", 5); ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/data-formats/changing-data-formats Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### KCQL Examples for Target Bucket and Path Source: https://docs.lenses.io/latest/connectors/~gitbook/pdf Provides practical examples of KCQL statements specifying different target bucket and path configurations, including optional paths and partitioning. ```SQL INSERT INTO testcontainer:pathToWriteTo SELECT * FROM topicA; ``` ```SQL INSERT INTO testcontainer SELECT * FROM topicA; ``` ```SQL INSERT INTO testcontainer:path/To/Write/To SELECT * FROM topicA PARTITIONBY fieldA; ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/complex-types/unwrapping-complex-types Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Kafka Console Producer for Source Cluster Source: https://docs.lenses.io/latest/k2k/getting-started/run-a-quick-example Executes a Kafka console producer within the source Kafka Docker container to send messages to the 'my-topic'. It connects to the specified bootstrap server and configures key parsing. ```bash docker compose exec sourceKafka \ /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9094 \ --topic my-topic \ --property parse.key=true \ --property key.separator=, ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/aggregations/aggregating-streams Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/getting-started Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/lpad Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/viewing-topic-metrics Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/rpad Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Provisioning Example with Kafka and HQ Connections Source: https://docs.lenses.io/latest/deployment/installation/docker/agent This YAML snippet demonstrates a minimal provisioning file with essential connections to Kafka and Lenses HQ, including agent key and server details. ```yaml lensesHq: - configuration: agentKey: value: ${LENSESHQ_AGENT_KEY} port: value: 10000 server: value: hq name: lenses-hq tags: - hq version: 1 kafka: - configuration: kafkaBootstrapServers: value: - PLAINTEXT://kafka:9092 protocol: value: PLAINTEXT name: kafka tags: - demo version: 1 ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/downloading-messages Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Full Lenses HQ Configuration Example Source: https://docs.lenses.io/latest/deployment/installation/linux/hq This is a comprehensive example of a final config.yaml file for Lenses HQ, including authentication, HTTP, agents, database, license, and logger configurations. It demonstrates a complete setup for running the application. ```yaml auth: users: - username: admin password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG # bcrypt("correcthorsebatterystaple"). administrators: - admin - [email protected] saml: enabled: true metadata: |- ... ... baseURL: https://example.com entityID: https://example.com userCreationMode: sso groupMembershipMode: sso http: address: ":8080" accessControlAllowOrigin: - https://example.com agents: address: ":10000" database: host: postgres:5432 username: panoptes password: password database: panoptes schema: insert-schema-here params: sslmode: require license: key: license_key_* acceptEULA: true logger: mode: text level: debug ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/metadata-fields Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Kafka Console Consumer for Source Cluster Source: https://docs.lenses.io/latest/k2k/getting-started/run-a-quick-example Executes a Kafka console consumer within the source Kafka Docker container to read messages from the 'my-topic'. It connects to the specified bootstrap server and configures key printing. ```bash docker compose exec sourceKafka \ /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9094 \ --topic my-topic \ --property print.key=true ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/trim Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/applications/sql-processors/time-and-windows Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/regexp Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Docker Compose for K2K and Kafka Source: https://docs.lenses.io/latest/k2k/getting-started/run-a-quick-example Defines the Docker Compose configuration for the K2K service and two Kafka clusters (source and target). It specifies Docker images, volumes, environment variables, and ports for each service. ```yaml services: k2k: image: "lensting/k2k:0.0.2-alpha" volumes: - ".:/pipelines" environment: K2K_PIPELINE_FOLDER: "/pipelines" OTEL_SERVICE_NAME: "k2k" OTEL_METRICS_EXPORTER: console OTEL_TRACES_EXPORTER: none OTEL_LOGS_EXPORTER: none command: - k2k - start - -f - /pipelines/k2k-pipeline.yml - -t - --acceptEULA sourceKafka: image: "apache/kafka:3.8.0" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9094,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: INTERNAL://sourceKafka:9092,EXTERNAL://127.0.0.1:9094 KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_NUM_PARTITIONS: 3 ports: - "9094:9094" targetKafka: image: "apache/kafka:3.8.0" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9099,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: INTERNAL://targetKafka:9092,EXTERNAL://127.0.0.1:9099 KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_NUM_PARTITIONS: 3 ports: - 9099:9099 ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/mkstring Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/applications/sql-processors/joins Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/searching-for-messages Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Deploy Lenses Agent on Linux Source: https://docs.lenses.io/latest/getting-started/quickstart/hands-on-walk-through-of-community-edition Instructions for deploying the Lenses Agent component on a Linux system. This covers downloading and running the agent. ```Bash wget https://packages.lenses.io/lenses-agent.tar.gz tar -xzf lenses-agent.tar.gz cd lenses-agent ./bin/lenses-agent start ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/indexof Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/applications/sql-processors/settings Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/regex_matches Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/iam/example-policies Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/upper Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/supported-data-formats Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/getting-started/connecting-lenses-to-your-environment/overview Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/string/base64 Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### KCQL Examples for Target Bucket and Path Source: https://docs.lenses.io/latest/connectors/kafka-connectors/sinks/aws-s3 Provides practical examples of KCQL statements specifying different target bucket and path configurations, including cases with and without a path prefix, and with partitioning. ```SQL INSERT INTO testbucket:pathToWriteTo SELECT * FROM topicA; ``` ```SQL INSERT INTO testbucket SELECT * FROM topicA; ``` ```SQL INSERT INTO testbucket:path/To/Write/To SELECT * FROM topicA PARTITIONBY fieldA; ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/topics/environment-topic-catalogue/approval-requests Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/obfuscation/first3 Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/nulls/as_nullable Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/applications/sql-processors Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Start Lenses Agent Source: https://docs.lenses.io/latest/deployment/installation/linux/agent Instructions on how to start the Lenses agent from the terminal. It shows the basic command and how to specify a configuration file. ```bash bin/lenses ``` ```bash bin/lenses lenses.conf ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/data-formats/changing-the-shape-of-data Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Example values.yaml for Lenses Agent Configuration Source: https://docs.lenses.io/latest/deployment/installation/helm/agent This YAML configuration file demonstrates how to set up the Lenses Agent. It includes settings for PostgreSQL storage, HQ agent key generation, and provisioning connections for Lenses HQ and Kafka clusters, including Kafka bootstrap servers and metrics configuration. ```yaml lensesAgent: storage: postgres: enabled: true host: [postgres.url] port: 5432 username: postgres password: changeMe database: agent hq: agentKey: secret: type: "createNew" name: "agentKey" value: "agent_key_*" provision: path: /mnt/provision-secrets connections: lensesHq: - name: lenses-hq version: 1 tags: ['hq'] configuration: server: value: hq-tls-test-lenses-hq.hq-agent-test.svc.cluster.local port: value: 10000 agentKey: value: ${LENSESHQ_AGENT_KEY} kafka: # There can only be one Kafka cluster at a time - name: kafka version: 1 tags: ['staging', 'pseudo-data-only'] configuration: kafkaBootstrapServers: value: - PLAINTEXT://kafka-1.svc.cluster.local:9092 - PLAINTEXT://kafka-2.svc.cluster.local:9092 protocol: value: PLAINTEXT # Metrics are strongly suggested for better Kafka cluster observability metricsType: value: JMX metricsPort: value: 9581 ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/aggregations Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/date-and-time/minute Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-studio/filter-by-timestamp-or-offset Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Kafka Console Consumer for Target Cluster Source: https://docs.lenses.io/latest/k2k/getting-started/run-a-quick-example Executes a Kafka console consumer within the target Kafka Docker container to read messages from the 'my-topic'. It connects to the specified bootstrap server and configures key printing. ```bash docker compose exec targetKafka \ /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9099 \ --topic my-topic \ --property print.key=true ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/numeric/min Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/environments/create-new-environment Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/applications/sql-processors/storage-format Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/obfuscation/first2 Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/applications/sql-processors/nullibility Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/monitoring-and-alerting Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/nulls Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Install Lenses Community Edition with Helm Source: https://docs.lenses.io/latest/user-guide/sql-reference/functions/obfuscation/first4 Instructions for deploying Lenses Community Edition using Helm charts on Kubernetes. This guide covers the necessary steps and configurations for a successful installation. ```Helm helm install lenses-io/lenses-community-edition --name my-lenses ``` -------------------------------- ### Deploy Lenses Agent on Linux Source: https://docs.lenses.io/latest/user-guide/tutorials/sql-processors/filtering-and-joins/using-multiple-topics Instructions for deploying the Lenses Agent component on a Linux system. This covers downloading and running the agent. ```Bash wget https://packages.lenses.io/lenses-agent.tar.gz tar -xzf lenses-agent.tar.gz cd lenses-agent ./bin/lenses-agent start ```