### Install KMinion (Basic)

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Basic installation command for KMinion using Helm.

```bash
# Basic installation
helm install my-kminion redpanda/kminion
```

--------------------------------

### Install Redpanda Connect

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Install Redpanda Connect using Helm. Supports basic installation, custom values files, specific namespaces, inline value overrides, and upgrading existing releases.

```bash
# Basic installation with default values
helm install my-connect redpanda/connect

# Install with custom values file
helm install my-connect redpanda/connect -f values.yaml

# Install in a specific namespace
helm install my-connect redpanda/connect --namespace streaming --create-namespace

# Install with inline value overrides
helm install my-connect redpanda/connect \
  --set deployment.replicaCount=3 \
  --set autoscaling.enabled=true \
  --set autoscaling.minReplicas=2 \
  --set autoscaling.maxReplicas=10

# Upgrade an existing release
helm upgrade my-connect redpanda/connect -f values.yaml

# Uninstall
helm uninstall my-connect
```

--------------------------------

### Install KMinion with Kafka Broker Configuration

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Install KMinion with specific Kafka broker addresses configured.

```bash
# Install with Kafka broker configuration
helm install my-kminion redpanda/kminion \
  --set kminion.config.kafka.brokers[0]="redpanda-0.redpanda:9092"
```

--------------------------------

### Install Redpanda Connectors via Helm

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Commands for installing the Redpanda Connectors chart with various configuration options.

```bash
# Basic installation
helm install my-connectors redpanda/connectors

# Install with broker configuration
helm install my-connectors redpanda/connectors \
  --set connectors.bootstrapServers="redpanda-0.redpanda:9092,redpanda-1.redpanda:9092" \
  --set connectors.groupID="my-connect-cluster"

# Install with custom values
helm install my-connectors redpanda/connectors -f connectors-values.yaml
```

--------------------------------

### Install KMinion with Custom Values

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Install the KMinion Helm chart using a custom values file. Ensure the values file is correctly specified.

```bash
helm install my-kminion redpanda/kminion -f kminion-values.yaml
```

--------------------------------

### Configure KMinion with Autoscaling

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Example YAML configuration for KMinion deployment including custom labels, resource autoscaling, and pod anti-affinity rules.

```yaml
replicaCount: 2

customLabels:
  team: platform
  environment: production

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70

podDisruptionBudget:
  maxUnavailable: 1

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - kminion
          topologyKey: kubernetes.io/hostname
```

--------------------------------

### Helm Commands for Migration

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/MIGRATION_FROM_BENTHOS.md

These commands demonstrate how to list existing Helm releases, add the Redpanda chart repository, install the new Redpanda Connect chart, and uninstall the old Benthos chart.

```bash
$ helm list -q
benthos-1730406308
```

```bash
$ helm repo add redpanda https://charts.redpanda.com/

```

```bash
$ helm install --generate-name -f values.yml redpanda/connect
...
```

```bash
$ helm uninstall benthos-1730406308
```

--------------------------------

### Redpanda Connect with Autoscaling

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure horizontal pod autoscaling for Redpanda Connect. This example shows a values.yaml snippet for enabling autoscaling based on CPU utilization or custom metrics.

```yaml
yaml

# values.yaml for autoscaling
deployment:
  replicaCount: 2

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80 # Uncomment to enable memory-based autoscaling
```

--------------------------------

### Enter development shell

Source: https://github.com/redpanda-data/helm-charts/blob/main/CONTRIBUTING.md

Commands to initialize the development environment using Nix or Task.

```sh
nix develop # Enter a development shell.
task shell # Or enter through task
nix develop -c fish  # Enter a development shell using fish/zsh
nix develop -c zsh  # Enter a development shell using fish/zsh
```

--------------------------------

### Enable Nix experimental features

Source: https://github.com/redpanda-data/helm-charts/blob/main/CONTRIBUTING.md

Appends the required experimental features to the Nix configuration file to enable flakes and nix-command.

```bash
# Or open the file in an editor and paste in this line.
# If you're using nix to manage your nix install, you'll have to find your own path :)
echo 'experimental-features = nix-command flakes' >> /etc/nix/nix.conf
```

--------------------------------

### Migrate Benthos to Redpanda Connect

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Step-by-step CLI commands to transition from legacy Benthos Helm releases to the Redpanda Connect chart.

```bash
# Step 1: List existing Benthos releases
helm list -q | grep benthos

# Step 2: Add Redpanda repository
helm repo add redpanda https://charts.redpanda.com/
helm repo update

# Step 3: Install Redpanda Connect alongside existing Benthos
# Note: The default root_path changed from /benthos to /redpanda-connect
helm install my-connect redpanda/connect -f values.yaml

# Step 4: Verify new deployment is healthy
kubectl get pods -l app.kubernetes.io/name=connect
kubectl logs -l app.kubernetes.io/name=connect

# Step 5: Update any external references to use new service
# Service names changed from *-benthos to *-redpanda-connect

# Step 6: Uninstall old Benthos release once verified
helm uninstall benthos-old-release
```

--------------------------------

### Redpanda Connect Basic Configuration

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure a basic Redpanda Connect pipeline using a `values.yaml` file. This includes setting deployment replicas, image details, resource limits, HTTP server configuration, and defining input, pipeline, and output components.

```yaml
# values.yaml for Redpanda Connect
deployment:
  replicaCount: 2
  terminationGracePeriodSeconds: 60
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "4195"

image:
  repository: docker.redpanda.com/redpandadata/connect
  pullPolicy: IfNotPresent
  tag: "4.68.0"

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

http:
  enabled: true
  address: "0.0.0.0:4195"
  root_path: "/redpanda-connect"
  debug_endpoints: false

# Pipeline configuration
config:
  input:
    kafka:
      addresses:
        - redpanda-0.redpanda.default.svc.cluster.local:9092
      topics:
        - input-topic
      consumer_group: connect-consumer

  pipeline:
    processors:
      - mapping: |
          root = this
          root.processed_at = now()
          root.uppercase_field = this.field.uppercase()

  output:
    kafka:
      addresses:
        - redpanda-0.redpanda.default.svc.cluster.local:9092
      topic: output-topic
```

--------------------------------

### Add Redpanda Helm Repository

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Add the Redpanda Helm repository to your local Helm configuration to access available charts. Update the repository cache and search for charts.

```bash
helm repo add redpanda https://charts.redpanda.com/

helm repo update

helm search repo redpanda

helm search repo redpanda/connect --versions
helm search repo redpanda/connectors --versions
helm search repo redpanda/kminion --versions
```

--------------------------------

### Configure Redpanda Monitoring Settings

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Enable monitoring by setting `enabled` to `true`. This creates a PodMonitor for Prometheus or VictoriaMetrics operators.

```json
{"annotations":{},"enabled":false,"labels":{},"namespaceSelector":{"any":true},"scrapeInterval":"30s"}
```

--------------------------------

### Configure Redpanda Docker Image Settings

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Set the Docker image repository, tag, and pull policy for Redpanda. Ensure the tag is set appropriately for your deployment.

```json
{"pullPolicy":"IfNotPresent","repository":"docker.redpanda.com/redpandadata/connectors","tag":""}
```

--------------------------------

### Configure Container Resources

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Defines CPU and memory limits and requests, along with the Java maximum heap size.

```json
{"javaMaxHeapSize":"2G","limits":{"cpu":"1","memory":"2350Mi"},"request":{"cpu":"1","memory":"2350Mi"}}
```

--------------------------------

### Configure Autoscaling and Resources

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Defines replica counts, CPU/memory resource limits, and horizontal pod autoscaling metrics for Redpanda Connect.

```yaml
deployment:
  replicaCount: 2

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

podDisruptionBudget:
  enabled: true
  minAvailable: 1

resources:
  limits:
    cpu: "1"
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 512Mi

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: connect
```

--------------------------------

### Set Size Limit for EmptyDir Volume

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Defines the size limit for an `emptyDir` volume. Defaults to `5Mi`.

```text
"5Mi"
```

--------------------------------

### Manage Connector Instances via REST API

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Operational commands for interacting with the Kafka Connect REST API to manage plugins, configurations, and connector lifecycle states.

```bash
# Port-forward to access the REST API
kubectl port-forward svc/my-connectors 8083:8083

# List installed connector plugins
curl -s http://localhost:8083/connector-plugins | jq

# Create a new connector
curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d '{
    "name": "file-source",
    "config": {
      "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
      "tasks.max": "1",
      "file": "/tmp/input.txt",
      "topic": "file-topic"
    }
  }'

# List all connectors
curl -s http://localhost:8083/connectors | jq

# Get connector status
curl -s http://localhost:8083/connectors/file-source/status | jq

# Get connector configuration
curl -s http://localhost:8083/connectors/file-source/config | jq

# Update connector configuration
curl -X PUT http://localhost:8083/connectors/file-source/config \
  -H "Content-Type: application/json" \
  -d '{
    "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "tasks.max": "2",
    "file": "/tmp/input.txt",
    "topic": "file-topic"
  }'

# Pause a connector
curl -X PUT http://localhost:8083/connectors/file-source/pause

# Resume a connector
curl -X PUT http://localhost:8083/connectors/file-source/resume

# Restart a connector
curl -X POST http://localhost:8083/connectors/file-source/restart

# Delete a connector
curl -X DELETE http://localhost:8083/connectors/file-source
```

--------------------------------

### Connectors with Monitoring Enabled

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Enable Prometheus monitoring for Kafka Connect metrics using PodMonitor. Configure scrape interval, labels, and annotations.

```yaml
# connectors-values.yaml with monitoring
monitoring:
  enabled: true
  scrapeInterval: 30s
  labels:
    release: prometheus
  annotations:
    prometheus.io/scrape: "true"
  namespaceSelector:
    any: true

service:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9404"
  ports:
    - name: prometheus
      port: 9404

container:
  javaGCLogEnabled: "true"
```

--------------------------------

### KMinion with SASL Authentication

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure KMinion with SASL authentication for secure Kafka/Redpanda clusters. This requires setting up environment variables for credentials.

```yaml
# kminion-values.yaml with SASL
kminion:
  config:
    kafka:
      brokers:
        - redpanda:9092
      clientId: kminion
      sasl:
        enabled: true
        mechanism: SCRAM-SHA-512
        username: kminion-user
        password: "${KAFKA_SASL_PASSWORD}"

deployment:
  env:
    secretKeyRefs:
      - name: KAFKA_SASL_PASSWORD
        secretName: kafka-credentials
        secretKey: password

```

--------------------------------

### Configure Service Account Management

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Manage ServiceAccount creation and configuration, including annotations and token mounting.

```json
{"annotations":{},"automountServiceAccountToken":false,"create":false,"name":""}
```

--------------------------------

### Configure Redpanda Version Tag

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Specify the Redpanda version to use. Refer to DockerHub for available stable and unstable tags.

```text
"Chart.appVersion"
```

--------------------------------

### Set Storage Medium for EmptyDir Volume

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Specifies the medium for an `emptyDir` volume. Defaults to `Memory`.

```text
"Memory"
```

--------------------------------

### Advanced Deployment Configuration for Connectors

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure advanced deployment settings including rolling updates, pod anti-affinity, node affinity, topology spread, node selectors, tolerations, security context, and environment variables.

```yaml
# connectors-values.yaml with advanced deployment settings
deployment:
  create: true
  strategy:
    type: RollingUpdate

  progressDeadlineSeconds: 600
  revisionHistoryLimit: 10
  terminationGracePeriodSeconds: 30

  # Pod anti-affinity for high availability
  podAntiAffinity:
    type: hard
    topologyKey: kubernetes.io/hostname
    weight: 100

  # Node affinity for specific node pools
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: workload-type
              operator: In
              values:
                - streaming

  # Topology spread across zones
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule

  # Node selector
  nodeSelector:
    node-type: kafka-connect

  # Tolerations
  tolerations:
    - key: dedicated
      operator: Equal
      value: kafka-connect
      effect: NoSchedule

  # Security context
  securityContext:
    fsGroup: 101
    runAsUser: 101
    fsGroupChangePolicy: OnRootMismatch

  # Environment variables
  extraEnv:
    - name: CONNECT_LOG4J_LOGGERS
      value: "org.apache.kafka.connect.runtime.rest=WARN"

  extraEnvFrom:
    - configMapRef:
        name: connector-env-config
```

--------------------------------

### KMinion with End-to-End Monitoring

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Enable end-to-end latency monitoring by configuring KMinion to produce and consume test messages. This includes settings for probe interval, topic management, producer, and consumer SLAs.

```yaml
# kminion-values.yaml with end-to-end monitoring
kminion:
  config:
    kafka:
      brokers:
        - redpanda:9092
      clientId: kminion

    minion:
      consumerGroups:
        enabled: true
        scrapeMode: adminApi
        granularity: partition

      topics:
        enabled: true
        granularity: partition

      logDirs:
        enabled: true

      endToEnd:
        enabled: true
        probeInterval: 100ms
        topicManagement:
          enabled: true
          name: kminion-end-to-end
          reconciliationInterval: 10m
          replicationFactor: 3
          partitionsPerBroker: 1
        producer:
          ackSla: 5s
          requiredAcks: all
        consumer:
          groupIdPrefix: kminion-e2e
          deleteStaleConsumerGroups: true
          roundtripSla: 20s
          commitSla: 10s

```

--------------------------------

### KMinion Basic Configuration

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure KMinion to connect to your Kafka/Redpanda cluster and expose Prometheus metrics. This includes image details, service type, resource limits, and basic Kafka/Minion settings.

```yaml
# kminion-values.yaml
replicaCount: 1

image:
  repository: redpandadata/kminion
  pullPolicy: IfNotPresent
  tag: "v2.3.0"

service:
  type: ClusterIP
  port: 8080
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/metrics"

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

kminion:
  config:
    kafka:
      brokers:
        - redpanda-0.redpanda.default.svc.cluster.local:9092
        - redpanda-1.redpanda.default.svc.cluster.local:9092
        - redpanda-2.redpanda.default.svc.cluster.local:9092
      clientId: kminion

    minion:
      consumerGroups:
        enabled: true
        scrapeMode: adminApi
        granularity: partition
        allowedGroups: [".*"]
        ignoredGroups: []

      topics:
        enabled: true
        granularity: partition
        allowedTopics: [".*"]
        ignoredTopics: ["^_.*"]
        infoMetric:
          configKeys: ["cleanup.policy", "retention.ms"]

      logDirs:
        enabled: true

    exporter:
      namespace: kminion
      host: ""
      port: 8080

    logger:
      level: info

```

--------------------------------

### KMinion with TLS

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure KMinion with TLS encryption for secure broker connections. This involves specifying certificate file paths and enabling TLS.

```yaml
# kminion-values.yaml with TLS
kminion:
  config:
    kafka:
      brokers:
        - redpanda:9093
      clientId: kminion
      tls:
        enabled: true
        caFilepath: /etc/tls/ca.crt
        certFilepath: /etc/tls/tls.crt
        keyFilepath: /etc/tls/tls.key
        insecureSkipTlsVerify: false

deployment:
  volumes:
    secrets:
      - secretName: kafka-tls-certs
        mountPath: /etc/tls

```

--------------------------------

### Configure Liveness Probe

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Sets the health check parameters for the container, including thresholds and delays.

```json
{"failureThreshold":3,"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1}
```

--------------------------------

### Specify Redpanda Docker Image Repository

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Set the Docker repository for pulling the Redpanda Docker image.

```text
"docker.redpanda.com/redpandadata/connectors"
```

--------------------------------

### KMinion with Custom Labels and Autoscaling

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure KMinion with custom labels and horizontal pod autoscaling. This section is intended for further configuration details not fully provided in the source.

```yaml

```

--------------------------------

### Configure Secrets and Environment Variables

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Injects sensitive credentials from Kubernetes Secrets and mounts TLS certificates for secure Kafka connections.

```yaml
# values.yaml with secrets
env:
  - name: KAFKA_SASL_USERNAME
    valueFrom:
      secretKeyRef:
        name: kafka-credentials
        key: username
  - name: KAFKA_SASL_PASSWORD
    valueFrom:
      secretKeyRef:
        name: kafka-credentials
        key: password
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: access-key-id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: secret-access-key

envFrom:
  - secretRef:
      name: additional-secrets
  - configMapRef:
      name: additional-config

extraVolumes:
  - name: tls-certs
    secret:
      secretName: kafka-tls-certs

extraVolumeMounts:
  - name: tls-certs
    mountPath: /etc/tls
    readOnly: true

config:
  input:
    kafka:
      addresses: ["redpanda:9092"]
      topics: ["secure-topic"]
      tls:
        enabled: true
        root_cas_file: /etc/tls/ca.crt
      sasl:
        mechanism: SCRAM-SHA-512
        user: "${KAFKA_SASL_USERNAME}"
        password: "${KAFKA_SASL_PASSWORD}"
```

--------------------------------

### Configure Service Management

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Manage service settings, including annotations, name, and ports. The default includes a port for Prometheus metrics.

```json
{"annotations":{},"name":"","ports":[{"name":"prometheus","port":9404}]}
```

--------------------------------

### Configure Container Security Context

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Sets security constraints for the container, such as privilege escalation policies.

```json
{"allowPrivilegeEscalation":false}
```

--------------------------------

### Configure Redpanda Logging Level

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Set the log level for Redpanda. Options range from `error` (least verbose) to `debug` (most verbose).

```json
{"level":"warn"}
```

--------------------------------

### Configure Prometheus Monitoring

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Enables ServiceMonitor and pod annotations for scraping metrics from Redpanda Connect.

```yaml
# values.yaml with Prometheus monitoring
deployment:
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "4195"
    prometheus.io/path: "/redpanda-connect/metrics"

serviceMonitor:
  enabled: true
  interval: "15s"
  scheme: http
  # For authenticated metrics endpoints
  # basicAuth:
  #   username: metrics-user
  #   password: secret-password

http:
  enabled: true
  address: "0.0.0.0:4195"
  debug_endpoints: true

# Optional: Enable telemetry
telemetry: true
```

--------------------------------

### Basic connectors-values.yaml Configuration

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Default configuration for Redpanda Connectors, specifying image details, broker addresses, REST API port, group ID, schema registry URL, producer settings, internal topic configuration, resource requests/limits, and probe settings.

```yaml
image:
  repository: docker.redpanda.com/redpandadata/connectors
  tag: "v1.0.31"
  pullPolicy: IfNotPresent

connectors:
  # Kafka/Redpanda broker addresses
  bootstrapServers: "redpanda-0.redpanda.default.svc.cluster.local:9092,redpanda-1.redpanda.default.svc.cluster.local:9092"

  # REST API port for connector management
  restPort: 8083

  # Unique cluster identifier
  groupID: "connectors-cluster"

  # Schema Registry URL (optional)
  schemaRegistryURL: "http://schema-registry:8081"

  # Producer settings for better throughput
  producerBatchSize: 131072
  producerLingerMS: 1

  # Internal topic configuration
  storage:
    replicationFactor:
      offset: 3
      config: 3
      status: 3
    topic:
      offset: "_connect_offsets"
      config: "_connect_configs"
      status: "_connect_status"

container:
  resources:
    request:
      cpu: "1"
      memory: 2350Mi
    limits:
      cpu: "2"
      memory: 4Gi
    javaMaxHeapSize: 3G

deployment:
  livenessProbe:
    initialDelaySeconds: 30
    periodSeconds: 10
    failureThreshold: 3
  readinessProbe:
    initialDelaySeconds: 60
    periodSeconds: 10
    successThreshold: 1

logging:
  level: info
```

--------------------------------

### Configure Datadog Logs and Metrics for Redpanda

Source: https://github.com/redpanda-data/helm-charts/blob/main/contrib/datadog-annotations/README.md

Use these annotations in your `values.yaml` to configure Datadog's log collection and metrics scraping for Redpanda. Ensure the `openmetrics_endpoint` is correctly set to your Redpanda instance's metrics endpoint.

```yaml
statefulset:
  annotations:
    ad.datadoghq.com/redpanda.logs: '[{"source": "redpanda", "service": "redpanda_cluster"}]'
    ad.datadoghq.com/redpanda.check_names: '["openmetrics", "openmetrics"]'
    ad.datadoghq.com/redpanda.init_configs: '[{}, {}]'
    ad.datadoghq.com/redpanda.instances: |-
      [
        {
          "openmetrics_endpoint": "http://%%host%%:9644/metrics",
          "namespace": "redpanda",
          "metrics": [
            {
              "vectorized_application_uptime": "application.uptime",
              "vectorized_application_build": "application.build"
            },
            {
              "vectorized_cluster_partition_committed_offset": "cluster.partition_committed_offset",
              "vectorized_cluster_partition_end_offset": "cluster.partition_end_offset",
              "vectorized_cluster_partition_high_watermark": "cluster.partition_high_watermark",
              "vectorized_cluster_partition_last_stable_offset": "cluster.partition_last_stable_offset",
              "vectorized_cluster_partition_leader": "cluster.partition_leader",
              "vectorized_cluster_partition_leader_id": "cluster.partition_leader_id",
              "vectorized_cluster_partition_records_fetched": "cluster.partition_records_fetched",
              "vectorized_cluster_partition_records_produced": "cluster.partition_records_produced",
              "vectorized_cluster_partition_under_replicated_replicas": "cluster.partition_under_replicated_replicas"
            },
            {
              "vectorized_httpd_connections_current": "httpd.connections_current",
              "vectorized_httpd_connections": "httpd.connections",
              "vectorized_httpd_read_errors": "httpd.read_errors",
              "vectorized_httpd_reply_errors": "httpd.reply_errors",
              "vectorized_httpd_requests_served": "httpd.requests_served"
            },
            {
              "vectorized_kafka_fetch_sessions_cache_mem_usage_bytes": "kafka.fetch_sessions_cache_mem_usage_bytes",
              "vectorized_kafka_fetch_sessions_cache_sessions_count": "kafka.fetch_sessions_cache_sessions_count",
              "vectorized_kafka_latency_fetch_latency_us": "kafka.latency_fetch_latency_us",
              "vectorized_kafka_latency_produce_latency_us": "kafka.latency_produce_latency_us",
              "vectorized_kafka_rpc_active_connections": "kafka.rpc_active_connections",
              "vectorized_kafka_rpc_connection_close_errors": "kafka.rpc_connection_close_errors",
              "vectorized_kafka_rpc_connects": "kafka.rpc_connects",
              "vectorized_kafka_rpc_consumed_mem_bytes": "kafka.rpc_consumed_mem_bytes",
              "vectorized_kafka_rpc_corrupted_headers": "kafka.rpc_corrupted_headers",
              "vectorized_kafka_rpc_dispatch_handler_latency": "kafka.rpc_dispatch_handler_latency",
              "vectorized_kafka_rpc_max_service_mem_bytes": "kafka.rpc_max_service_mem_bytes",
              "vectorized_kafka_rpc_method_not_found_errors": "kafka.rpc_method_not_found_errors",
              "vectorized_kafka_rpc_received_bytes": "kafka.rpc_received_bytes",
              "vectorized_kafka_rpc_requests_blocked_memory": "kafka.rpc_requests_blocked_memory",
              "vectorized_kafka_rpc_requests_completed": "kafka.rpc_requests_completed",
              "vectorized_kafka_rpc_requests_pending": "kafka.rpc_requests_pending",
              "vectorized_kafka_rpc_sent_bytes": "kafka.rpc_sent_bytes",
              "vectorized_kafka_rpc_service_errors": "kafka.rpc_service_errors",
              "vectorized_kafka_group_offset": "kafka.group_offset"
            },
            {
              "vectorized_leader_balancer_leader_transfer_error": "leader.balancer_leader_transfer_error",
              "vectorized_leader_balancer_leader_transfer_no_improvement": "leader.balancer_leader_transfer_no_improvement",
              "vectorized_leader_balancer_leader_transfer_succeeded": "leader.balancer_leader_transfer_succeeded",
              "vectorized_leader_balancer_leader_transfer_timeout": "leader.balancer_leader_transfer_timeout"
            },
            {
              "vectorized_pandaproxy_request_latency": "pandaproxy.request_latency"
            },
            {
              "vectorized_reactor_abandoned_failed_futures": "reactor.abandoned_failed_futures",
              "vectorized_reactor_aio_bytes_read": "reactor.aio_bytes_read",
              "vectorized_reactor_aio_bytes_write": "reactor.aio_bytes_write",
              "vectorized_reactor_aio_errors": "reactor.aio_errors",
              "vectorized_reactor_aio_reads": "reactor.aio_reads",
              "vectorized_reactor_aio_writes": "reactor.aio_writes",
              "vectorized_reactor_cpp_exceptions": "reactor.cpp_exceptions",
              "vectorized_reactor_cpu_busy_ms": "reactor.cpu_busy_ms",
              "vectorized_reactor_cpu_steal_time_ms": "reactor.cpu_steal_time_ms"
            }
          ]
        }
      ]

```

--------------------------------

### Configure Datadog OpenMetrics in values.yaml

Source: https://github.com/redpanda-data/helm-charts/blob/main/contrib/datadog-annotations/README.md

Add this annotation to your values.yaml file to enable Datadog monitoring for Redpanda metrics.

```yaml
statefulset:
  annotations:
    ad.datadoghq.com/redpanda.checks: |
      {
        "openmetrics": {
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:9644/metrics",
              "namespace": "redpanda",
              "max_returned_metrics": 99999,
              "request_size": 128,
              "metrics": [
                {
                  "vectorized_application_uptime": "application.uptime"
                },
                {
                  "vectorized_application_build": "application.build"
                },
                {
                  "vectorized_cluster_partition_committed_offset": "cluster.partition_committed_offset"
                },
                {
                  "vectorized_cluster_partition_end_offset": "cluster.partition_end_offset"
                },
                {
                  "vectorized_cluster_partition_high_watermark": "cluster.partition_high_watermark"
                },
                {
                  "vectorized_cluster_partition_last_stable_offset": "cluster.partition_last_stable_offset"
                },
                {
                  "vectorized_cluster_partition_leader": "cluster.partition_leader"
                },
                {
                  "vectorized_cluster_partition_leader_id": "cluster.partition_leader_id"
                },
                {
                  "vectorized_cluster_partition_records_fetched": "cluster.partition_records_fetched"
                },
                {
                  "vectorized_cluster_partition_records_produced": "cluster.partition_records_produced"
                },
                {
                  "vectorized_cluster_partition_under_replicated_replicas": "cluster.partition_under_replicated_replicas"
                },
                {
                  "vectorized_httpd_connections_current": "httpd.connections_current"
                },
                {
                  "vectorized_httpd_connections": "httpd.connections"
                },
                {
                  "vectorized_httpd_read_errors": "httpd.read_errors"
                },
                {
                  "vectorized_httpd_reply_errors": "httpd.reply_errors"
                },
                {
                  "vectorized_httpd_requests_served": "httpd.requests_served"
                },
                {
                  "vectorized_kafka_fetch_sessions_cache_mem_usage_bytes": "kafka.fetch_sessions_cache_mem_usage_bytes"
                },
                {
                  "vectorized_kafka_fetch_sessions_cache_sessions_count": "kafka.fetch_sessions_cache_sessions_count"
                },
                {
                  "vectorized_kafka_latency_fetch_latency_us": "kafka.latency_fetch_latency_us"
                },
                {
                  "vectorized_kafka_latency_produce_latency_us": "kafka.latency_produce_latency_us"
                },
                {
                  "vectorized_kafka_rpc_active_connections": "kafka.rpc_active_connections"
                },
                {
                  "vectorized_kafka_rpc_connection_close_errors": "kafka.rpc_connection_close_errors"
                },
                {
                  "vectorized_kafka_rpc_connects": "kafka.rpc_connects"
                },
                {
                  "vectorized_kafka_rpc_consumed_mem_bytes": "kafka.rpc_consumed_mem_bytes"
                },
                {
                  "vectorized_kafka_rpc_corrupted_headers": "kafka.rpc_corrupted_headers"
                },
                {
                  "vectorized_kafka_rpc_dispatch_handler_latency": "kafka.rpc_dispatch_handler_latency"
                },
                {
                  "vectorized_kafka_rpc_max_service_mem_bytes": "kafka.rpc_max_service_mem_bytes"
                },
                {
                  "vectorized_kafka_rpc_method_not_found_errors": "kafka.rpc_method_not_found_errors"
                },
                {
                  "vectorized_kafka_rpc_received_bytes": "kafka.rpc_received_bytes"
                },
                {
                  "vectorized_kafka_rpc_requests_blocked_memory": "kafka.rpc_requests_blocked_memory"
                },
                {
                  "vectorized_kafka_rpc_requests_completed": "kafka.rpc_requests_completed"
                },
                {
                  "vectorized_kafka_rpc_requests_pending": "kafka.rpc_requests_pending"
                },
                {
                  "vectorized_kafka_rpc_sent_bytes": "kafka.rpc_sent_bytes"
                },
                {
                  "vectorized_kafka_rpc_service_errors": "kafka.rpc_service_errors"
                },
                {
                  "vectorized_kafka_group_offset": "kafka.group_offset"
                },
                {
```

--------------------------------

### Configure Ingress for Redpanda Connect

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Exposes the service externally via Kubernetes Ingress with TLS termination and CORS settings.

```yaml
# values.yaml with ingress configuration
service:
  type: ClusterIP
  port: 80
  targetPort: http

ingress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: connect.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: connect-tls
      hosts:
        - connect.example.com

http:
  enabled: true
  address: "0.0.0.0:4195"
  root_path: "/redpanda-connect"
  cors:
    enabled: true
    allowedOrigins:
      - "https://dashboard.example.com"
```

--------------------------------

### KMinion with ServiceMonitor

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure Prometheus ServiceMonitor for automatic scrape configuration discovery. This includes settings for additional labels, target labels, and relabeling/metric relabeling rules.

```yaml
# kminion-values.yaml with ServiceMonitor
serviceMonitor:
  create: true
  additionalLabels:
    release: prometheus-stack
  targetLabels: []
  honorLabels: false
  scrapeTimeout: 10s
  interval: 15s
  relabelings:
    - sourceLabels: [__meta_kubernetes_pod_label_app]
      separator: ;
      regex: (.*)
      targetLabel: app
      replacement: $1
      action: replace
  metricRelabelings:
    - sourceLabels: [__name__]
      regex: kminion_kafka_topic_partition.*
      action: keep

service:
  type: ClusterIP
  port: 8080
  annotations: {}

```

--------------------------------

### KMinion DaemonSet Mode

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Deploy KMinion as a DaemonSet for per-node monitoring in large clusters. This configuration disables replicaCount and enables daemonset mode with node selectors and tolerations.

```yaml
# kminion-values.yaml for DaemonSet mode
replicaCount: 0

daemonset:
  enabled: true

nodeSelector:
  kafka-node: "true"

tolerations:
  - key: dedicated
    operator: Equal
    value: kafka
    effect: NoSchedule

kminion:
  config:
    kafka:
      brokers:
        - redpanda:9092
      rackId: "${NODE_NAME}"

```

--------------------------------

### Set Storage Volume Mount Path

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Specifies the mount path for the storage volume.

```text
"/tmp"
```

--------------------------------

### Retrieve Application Access URL

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/kminion/templates/NOTES.txt

Conditional logic to determine the application URL based on the configured Kubernetes service type.

```helm
{{- if .Values.ingress.enabled }}
     {{- range .Values.ingress.hosts }}
     http://{{ . }}
     {{- end }}
{{- else if contains "NodePort" .Values.service.type }}
  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "kminion.fullname" . }})
  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "kminion.fullname" . }}'
  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "kminion.fullname" . }} --template "{{ "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}" }}")
  echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "kminion.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}
```

--------------------------------

### Configure Pod Anti-Affinity

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Defines rules for scheduling pods to ensure high availability across nodes or zones.

```json
{"custom":{},"topologyKey":"kubernetes.io/hostname","type":"hard","weight":100}
```

--------------------------------

### Set Redpanda Log Level

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Specify the desired log level. Valid values are `error`, `warn`, `info`, and `debug`.

```text
"warn"
```

--------------------------------

### Retrieve Redpanda Connect Application URL

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/templates/NOTES.txt

Conditional logic to determine the access method for Redpanda Connect based on the Helm ingress and service configuration.

```yaml
{{- if .Values.ingress.enabled }}
{{- range $host := .Values.ingress.hosts }}
  {{- range .paths }}
  http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
  {{- end }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "redpanda-connect.fullname" . }})
  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "redpanda-connect.fullname" . }}'
  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "redpanda-connect.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
  echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "redpanda-connect.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}
```

--------------------------------

### Connectors with SASL Authentication

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Configure SASL authentication for secure connections to Redpanda/Kafka clusters. Requires a Kubernetes Secret for credentials.

```yaml
# connectors-values.yaml with SASL
connectors:
  bootstrapServers: "redpanda:9092"
  groupID: "secure-connectors"

auth:
  sasl:
    enabled: true
    mechanism: scram-sha-512
    userName: connect-user
    secretRef: kafka-sasl-credentials

# Create the secret separately
---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-sasl-credentials
type: Opaque
stringData:
  password: "your-secure-password"
```

--------------------------------

### Set Redpanda Image Pull Policy

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Defines the image pull policy. Defaults to `Always` if `image.tag` is 'latest', otherwise `IfNotPresent`.

```text
"IfNotPresent"
```

--------------------------------

### Set Storage Volume Name

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Defines the name for the storage volume mount.

```text
"rp-connect-tmp"
```

--------------------------------

### Add Annotations to Service

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Specify custom annotations to be added to the Kubernetes Service.

```json
{}
```

--------------------------------

### Override Connectors Name

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Allows overriding the default `connectors.name` template.

```text
""
```

--------------------------------

### Configure Image Pull Secrets

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Provide credentials for private image repositories using Kubernetes pull secrets.

```yaml
[]
```

--------------------------------

### Connectors with TLS Encryption

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Enable TLS encryption for broker connections. Requires Kubernetes Secrets for CA, client certificate, and client key.

```yaml
# connectors-values.yaml with TLS
connectors:
  bootstrapServers: "redpanda:9093"
  groupID: "tls-connectors"

  brokerTLS:
    enabled: true
    ca:
      secretRef: kafka-tls-ca
      secretNameOverwrite: "ca.crt"
    cert:
      secretRef: kafka-tls-client
      secretNameOverwrite: "tls.crt"
    key:
      secretRef: kafka-tls-client
      secretNameOverwrite: "tls.key"

# Create TLS secrets
---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-tls-ca
type: Opaque
data:
  ca.crt: <base64-encoded-ca-certificate>

---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-tls-client
type: kubernetes.io/tls
data:
  tls.crt: <base64-encoded-client-certificate>
  tls.key: <base64-encoded-client-key>
```

--------------------------------

### Control Service Account Token Automount

Source: https://github.com/redpanda-data/helm-charts/blob/main/charts/connectors/README.md

Determines whether the ServiceAccount should automatically mount API credentials.

```boolean
false
```

--------------------------------

### Redpanda Connect with Streams Mode

Source: https://context7.com/redpanda-data/helm-charts/llms.txt

Enable streams mode in Redpanda Connect for managing multiple pipelines via a REST API. This configuration includes enabling streams, specifying a ConfigMap for stream definitions, and enabling the API.

```yaml
# values.yaml for streams mode
deployment:
  replicaCount: 3
  rolloutConfigMap: true

streams:
  enabled: true
  streamsConfigMap: "my-streams-config"
  api:
    enable: true

http:
  enabled: true
  address: "0.0.0.0:4195"
  root_path: "/redpanda-connect"

# Create a ConfigMap with stream definitions
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-streams-config
data:
  stream1.yaml: |
    input:
      kafka:
        addresses: ["redpanda:9092"]
        topics: ["events"]
        consumer_group: stream1-group
    output:
      kafka:
        addresses: ["redpanda:9092"]
        topic: processed-events

  stream2.yaml: |
    input:
      http_server:
        path: /post
    pipeline:
      processors:
        - mapping: |
            root = this
            root.received_at = now()
    output:
      kafka:
        addresses: ["redpanda:9092"]
        topic: webhook-events
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.