### Install k3d Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Installs the k3d command-line tool, a wrapper around k3s in Docker, used for creating local Kubernetes clusters. This script downloads and executes the installation. ```shell wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/v5.4.6/install.sh | bash ``` -------------------------------- ### Install Redis with Helm Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Installs or upgrades a Redis instance using the Bitnami Helm chart with specified values and namespace. ```shell helm upgrade --install -f ./values-redis.yaml \ --namespace kpops edis bitnami/redis ``` -------------------------------- ### Install Kafka using Helm Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Installs Kafka, Zookeeper, Schema Registry, Kafka Rest Proxy, and Kafka Connect using the Confluent Helm chart. It deploys these components into the 'kpops' namespace and waits for the deployment to complete. ```shell helm upgrade \ --install \ --version 0.6.1 \ --values ./kafka.yaml \ --namespace kpops \ --create-namespace \ --wait \ k8kafka confluentinc/cp-helm-charts ``` -------------------------------- ### Redis Helm Chart Values Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Example configuration values for deploying a standalone Redis instance using the Bitnami Helm chart. ```yaml architecture: standalone auth: enabled: false master: count: 1 configuration: "databases 1" image: tag: 7.0.8 ``` -------------------------------- ### Install KPOps Package Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Installs the KPOps Python package using pip. This command downloads and installs the latest version of KPOps from the Python Package Index. ```python pip install kpops ``` -------------------------------- ### Kafka Helm Chart Values Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md An example configuration file (`kafka.yaml`) for the Confluent Kafka Helm chart. This configuration specifies minimal resources for deploying a single Kafka Broker, Schema Registry, Zookeeper, Kafka Rest Proxy, and Kafka Connect. ```yaml --8<-- ./docs/resources/setup/kafka.yaml --8<-- ``` -------------------------------- ### Initialize and Update Git Submodules Source: https://github.com/bakdata/kpops/blob/main/CONTRIBUTING.md Fetches the KPOps examples repository, which is included as a Git submodule. This command is essential for accessing example resources locally. ```bash git submodule init git submodule update --recursive ``` -------------------------------- ### Initialize and Update Git Submodules Source: https://github.com/bakdata/kpops/blob/main/docs/docs/developer/contributing.md Fetches the KPOps examples repository, which is included as a Git submodule. This command is essential for accessing example resources locally. ```bash git submodule init git submodule update --recursive ``` -------------------------------- ### Dockerfile for Kafka Connect JDBC Driver Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md A Dockerfile to create a custom Kafka Connect image that includes the Confluent JDBC driver. It starts from a Confluent Kafka Connect base image and installs the driver using the confluent-hub tool. ```dockerfile FROM confluentinc/cp-kafka-connect:7.1.3 RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.6.0 ``` -------------------------------- ### Deploy Streams Explorer using Helm Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Deploys the Streams Explorer application using Helm, specifying the version, values file, namespace, and release name. This command installs or upgrades the Streams Explorer deployment. ```shell helm upgrade \ --install \ --version 0.2.3 \ --values ./streams-explorer.yaml \ --namespace kpops \ streams-explorer streams-explorer/streams-explorer ``` -------------------------------- ### HelmApp Configuration Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/core-concepts/components/helm-app.md Example configuration file for HelmApp, demonstrating how to specify the Helm chart and values for deployment. ```yaml --8<-- ./docs/resources/pipeline-components/helm-app.yaml --8<-- ``` -------------------------------- ### ProducerApp Configuration Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/core-concepts/components/producer-app.md Example configuration for the ProducerApp using a pipeline.yaml file. This demonstrates how to set up a Kafka producer application. ```yaml --8< ./docs/resources/pipeline-components/producer-app.yaml --8< ``` -------------------------------- ### Install KPOps Source: https://github.com/bakdata/kpops/blob/main/README.md Installs the KPOps package using pip. This is the primary method for getting KPOps set up in your Python environment. ```sh pip install kpops ``` -------------------------------- ### StreamsApp Configuration Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/core-concepts/components/streams-app.md Example configuration file for a StreamsApp, demonstrating how to set up a Kafka Streams application. ```yaml --8< ./docs/resources/pipeline-components/streams-app.yaml --8< ``` -------------------------------- ### Streams Explorer Helm Chart Values Configuration Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md An example configuration file (`streams-explorer.yaml`) for the Streams Explorer Helm chart. It specifies the image tag, cluster settings, schema registry URL, Kafka Connect URL, and resource requests/limits. ```yaml imageTag: "v2.1.2" config: K8S__deployment__cluster: true SCHEMAREGISTRY__url: http://k8kafka-cp-schema-registry.kpops.svc.cluster.local:8081 KAFKACONNECT__url: http://k8kafka-cp-kafka-connect.kpops.svc.cluster.local:8083 resources: requests: cpu: 200m memory: 300Mi limits: cpu: 200m memory: 300Mi ``` -------------------------------- ### Serve kpops Documentation Source: https://github.com/bakdata/kpops/blob/main/docs/README.md Installs documentation dependencies and serves the documentation locally using the 'just' task runner. Access the docs at http://localhost:8000/. ```sh just serve-docs ``` -------------------------------- ### Deploy Word-count Pipeline Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Deploys the word-count pipeline using Kpops, executing the deployment configuration. ```shell kpops deploy word-count/pipeline.yaml --execute ``` -------------------------------- ### Uninstall Redis Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Uninstalls the Redis instance deployed via Helm from the 'kpops' namespace. ```shell helm --namespace kpops uninstall redis ``` -------------------------------- ### KafkaSourceConnector Configuration Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/core-concepts/components/kafka-source-connector.md Example configuration for a KafkaSourceConnector in a pipeline.yaml file. ```yaml --8< ./docs/resources/pipeline-components/kafka-source-connector.yaml --8< ``` -------------------------------- ### Helm Release Name Trimming Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/migration-guide/v2-v3.md Demonstrates how KPOps v3 handles long Helm release names by trimming them and appending a SHA-1 hash. This ensures compatibility with Helm's naming constraints. ```console example-component-name-too-long-fake-fakef-0a7fc ----> 53 chars ---------------------------------------------- ----- ^Shortened helm_release_name ^first 5 characters of SHA1(helm_release_name) ``` -------------------------------- ### Port Forward Kafka Connect Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Establishes a port forwarding connection to the Kafka Connect service in the 'kpops' namespace. ```shell kubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:8083 ``` -------------------------------- ### Add and Update Helm Repositories Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Adds the Bitnami Helm repository and updates the local repository cache to fetch the latest chart information. ```shell helm repo add bitnami https://charts.bitnami.com/bitnami && \ helm repo update ``` -------------------------------- ### Install PostgreSQL with Helm Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/examples/atm-fraud-pipeline.md Installs or upgrades a PostgreSQL instance using the Bitnami Helm chart, applying configurations from postgresql.yaml. ```shell helm upgrade --install -f ./postgresql.yaml \ --namespace kpops postgresql bitnami/postgresql ``` -------------------------------- ### Pipeline Configuration Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/resources/pipeline-components/pipeline.md This snippet shows a typical configuration for a data pipeline using YAML format. It outlines the structure for defining stages, components, and their respective configurations. ```yaml --8< ./docs/resources/pipeline-components/pipeline.yaml --8< ``` -------------------------------- ### Word-count Pipeline Configuration Source: https://github.com/bakdata/kpops/blob/main/docs/docs/resources/examples/pipeline.md This YAML file outlines the pipeline configuration for the Word-count example. It details how text data is processed to count word occurrences. ```yaml --8< https://raw.githubusercontent.com/bakdata/kpops-examples/main/word-count/pipeline.yaml --8< ``` -------------------------------- ### Port Forward Kafka REST Proxy Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Establishes a port forwarding connection to the Kafka REST proxy service in the 'kpops' namespace. ```shell kubectl port-forward --namespace kpops service/k8kafka-cp-rest 8082:8082 ``` -------------------------------- ### Port Forward Streams Explorer Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Establishes a port forwarding connection to the Streams Explorer service for accessing the pipeline visualization. ```shell kubectl port-forward -n kpops service/streams-explorer 8080:8080 ``` -------------------------------- ### Run Git Hooks and dprint Formatter Source: https://github.com/bakdata/kpops/blob/main/CONTRIBUTING.md Installs Git hooks for linting, formatting, and documentation generation. It also shows how to manually trigger all hooks and use the dprint formatter. ```bash lefthook install lefthook run pre-commit --all-files ``` -------------------------------- ### KafkaSinkConnector Configuration Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/core-concepts/components/kafka-sink-connector.md Example configuration for the KafkaSinkConnector in a pipeline.yaml file. This includes settings for connecting to Kafka and managing the sink. ```yaml --8< ./docs/resources/pipeline-components/kafka-sink-connector.yaml --8< ``` -------------------------------- ### Clean Word-count Pipeline Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/quick-start.md Removes the word-count pipeline using Kpops, with verbose output and execution enabled. ```shell kpops clean word-count/pipeline.yaml --verbose --execute ``` -------------------------------- ### Run Git Hooks and dprint Formatter Source: https://github.com/bakdata/kpops/blob/main/docs/docs/developer/contributing.md Installs Git hooks for linting, formatting, and documentation generation. It also shows how to manually trigger all hooks and use the dprint formatter. ```bash lefthook install lefthook run pre-commit --all-files ``` -------------------------------- ### Word-count Pipeline Defaults Source: https://github.com/bakdata/kpops/blob/main/docs/docs/resources/examples/defaults.md This snippet displays the default configuration for the Word-count Pipeline in KPOps. It outlines parameters for Kafka topics and basic pipeline settings. ```yaml --- # Default configuration for the Word-count pipeline # Kafka topic configuration kafka: bootstrap_servers: "kafka:9092" input_topic: "text-input" output_topic: "word-counts" # Pipeline specific configuration pipeline: word_separator: " " ``` -------------------------------- ### PostgreSQL Helm Chart Values Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/examples/atm-fraud-pipeline.md Example values for the PostgreSQL Helm chart, configuring authentication, database name, user credentials, and persistence settings. ```yaml auth: database: app_db enablePostgresUser: true password: AppPassword postgresPassword: StrongPassword username: app1 primary: persistence: enabled: false existingClaim: postgresql-data-claim volumePermissions: enabled: true ``` -------------------------------- ### Global Configuration for Components and Pipeline Base Directory Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/migration-guide/v2-v3.md Updates the configuration to define `components_module` and `pipeline_base_dir` globally in `config.yaml`. This replaces previous CLI parameters. ```diff kafka_brokers: "http://k8 Kafka-cp-kafka-headless.kpops.svc.cluster.local:9092" environment: development + components_module: components + pipeline_base_dir: pipelines ``` -------------------------------- ### Example of Variable Substitution Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/core-concepts/variables/substitution.md This example demonstrates how component-specific variables can be used within a pipeline definition. It shows referencing attributes of a component and its subattributes. ```yaml --8< ./docs/resources/variables/variable_substitution.yaml --8< ``` -------------------------------- ### ATM Fraud Pipeline Configuration Source: https://github.com/bakdata/kpops/blob/main/docs/docs/resources/examples/pipeline.md This YAML file defines the pipeline configuration for the ATM Fraud detection example. It specifies the data sources, processing steps, and output destinations for detecting fraudulent transactions. ```yaml --8< https://raw.githubusercontent.com/bakdata/kpops-examples/main/atm-fraud/pipeline.yaml --8< ``` -------------------------------- ### Create k3d Docker Registry Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Creates a local Docker registry using k3d, which is necessary for pushing custom Docker images when an existing Docker registry is not available. This registry will be accessible on a specified port. ```shell k3d registry create kpops-registry.localhost --port 12345 ``` -------------------------------- ### Clone kpops Repository Source: https://github.com/bakdata/kpops/blob/main/docs/README.md Clones the kpops project repository from GitHub. This is the first step to get the project code. ```sh git clone git@github.com:bakdata/kpops.git ``` -------------------------------- ### Build and Push Custom Kafka Connect Image Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Builds a Docker image with the specified Dockerfile and tags it for a local registry. It then pushes the image to the registry, making it available for Kubernetes deployments. ```shell docker build . --tag localhost:12345/kafka-connect-jdbc:7.1.3 && \ docker push localhost:12345/kafka-connect-jdbc:7.1.3 ``` -------------------------------- ### KPOps Generate Command Example Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/migration-guide/v3-v4.md Demonstrates how to generate a KPOps pipeline using the 'dev' environment and specifying a pipeline file within a distributed defaults structure. This command illustrates the new way of handling default configurations. ```bash kpops generate \ --environment dev \ ./pipelines/distributed-defaults/pipeline-deep/pipeline.yaml ``` -------------------------------- ### Add and Update Streams Explorer Helm Repository Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Adds the Streams Explorer Helm repository and updates the local Helm repository cache. This is a prerequisite for deploying Streams Explorer. ```shell helm repo add streams-explorer https://bakdata.github.io/streams-explorer && \ helm repo update ``` -------------------------------- ### Word-count pipeline.yaml Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/what-is-kpops.md An example pipeline definition for a word-count application using KPOps. This file specifies the components and configuration for deploying a Kafka pipeline. ```yaml apiVersion: kpops.com/v1alpha1 kind: Pipeline metadata: name: word-count spec: kafka: topic: partitions: 3 replicationFactor: 1 streams: - name: word-count-streams applicationId: word-count image: docker.io/bitnami/kafka-streams-example:2.8.0 processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams processor: - type: kafka-streams ``` -------------------------------- ### Add streams-bootstrap to defaults.yaml Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/migration-guide/v2-v3.md This snippet shows the addition of `streams-bootstrap` configuration to the `defaults.yaml` file, including `repo_config` and `version`. ```diff kafka-app: app: streams: ... + streams-bootstrap: repo_config: ... version: ... ``` -------------------------------- ### Add Confluent Helm Repository Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/getting-started/setup.md Adds the Confluent Kafka Helm chart repository to the Helm configuration and updates the local repository index. This allows Helm to fetch the necessary Kafka charts. ```shell helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/ && helm repo update ``` -------------------------------- ### kpops init Command Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/references/cli-commands.md Initializes a new KPOps project at a specified path. It can include optional configuration settings in the generated 'config.yaml'. ```APIDOC kpops init: Usage: $ kpops init [OPTIONS] PATH Arguments: PATH: Path for a new KPOps project. It should lead to an empty (or non-existent) directory. The part of the path that doesn't exist will be created. [required] Options: --config-include-optional / --no-config-include-optional: Whether to include non-required settings in the generated 'config.yaml' [default: no-config-include-optional] --help: Show this message and exit. ``` -------------------------------- ### KPOps API Manifest Removal Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/migration-guide/v8-v9.md Indicates the removal of the 'kpops.manifest()' API function. Resource manifesting is now handled through operation commands. ```python # The kpops.manifest() API has been removed. # Use operation modes with deploy, destroy, reset, or clean commands instead. ``` -------------------------------- ### Manifesting Resources with Operation Mode Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/migration-guide/v8-v9.md Demonstrates how to manifest resources by setting the operation mode via command-line argument or environment variable. This replaces the previous 'kpops manifest' command. ```bash # Using command-line argument kpops deploy --operation-mode manifest # Using environment variable export KPOPS_OPERATION_MODE=manifest kpops deploy ``` -------------------------------- ### Deploy ATM Fraud Detection Pipeline Source: https://github.com/bakdata/kpops/blob/main/docs/docs/user/examples/atm-fraud-pipeline.md Deploys the ATM fraud detection pipeline using KPOps. It requires cloning the repository, installing dependencies, setting environment variables, and executing the deployment. ```shell export DOCKER_REGISTRY=bakdata && \ export NAMESPACE=kpops kpops deploy atm-fraud/pipeline.yaml --execute ```