### Example Kubernetes Manifest URL for Bundle

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_installation_details.md

This URL format is used for applying the operator installation bundle for specific versions. Replace {OPERATOR_VERSION} with the specific version.

```bash
https://github.com/Altinity/clickhouse-operator/raw/{OPERATOR_VERSION}/deploy/operator/clickhouse-operator-install-bundle.yaml
```

--------------------------------

### Custom Deployment with Pod and VolumeClaim Templates

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

This example demonstrates a more complex setup with explicit pod and volume claim templates defined for a specific cluster. It includes custom container images and volume mounts.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "pv-log"
spec:
  configuration:
    clusters:
      - name: "deployment-pv"
        templates:
          podTemplate: pod-template-with-volumes
        layout:
          shardsCount: 2
          replicasCount: 2

  templates:
    podTemplates:
      - name: pod-template-with-volumes
        spec:
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:24.8
              volumeMounts:
                - name: data-storage-vc-template
                  mountPath: /var/lib/clickhouse
                - name: log-storage-vc-template
                  mountPath: /var/log/clickhouse-server

    volumeClaimTemplates:
      - name: data-storage-vc-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 3Gi
      - name: log-storage-vc-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 2Gi
```

--------------------------------

### Deploy Trivial ClickHouse Installation

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

Applies a basic ClickHouseInstallation resource to create a 1-shard, 1-replica ClickHouse cluster. WARNING: This example does not configure persistent storage and is not suitable for production.

```bash
kubectl apply -n test-clickhouse-operator -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/docs/chi-examples/01-simple-layout-01-1shard-1repl.yaml
```

```text
clickhouseinstallation.clickhouse.altinity.com/simple-01 created
```

--------------------------------

### Install ClickHouse from Manifest

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/chi_update_clickhouse_version.md

Apply the ClickHouse installation manifest to deploy a new ClickHouse cluster.

```bash
kubectl -n dev apply -f 08-clickhouse-version-update-01-initial-position.yaml
```

--------------------------------

### Enable Docker Multi-arch Support

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/devspace.md

Install QEMU and enable user-static support for building multi-architecture Docker images.

```bash
sudo apt-get install -y qemu binfmt-support qemu-user-static
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
```

--------------------------------

### Install ClickHouse Operator with Custom Namespace

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

Use the installer script to customize the installation, such as specifying the operator's namespace. The script downloads necessary files and installs the operator.

```bash
curl -s https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator-web-installer/clickhouse-operator-install.sh | OPERATOR_NAMESPACE=test-clickhouse-operator bash
```

--------------------------------

### Install ClickHouse Operator (v1beta1)

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

Use this command to install the operator on Kubernetes versions prior to 1.17 in the kube-system namespace. It applies a specific installation bundle.

```bash
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle-v1beta1.yaml
```

--------------------------------

### Use ClickHouseInstallationTemplate in Installation

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_configuration.md

Shows how to reference a ClickHouseInstallationTemplate within a ClickHouseInstallation resource.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
...
spec:
  useTemplates:
    - name: clickhouse-stable
...
```

--------------------------------

### FIPS Startup Banner Example

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/fips.md

This log output indicates the FIPS build and runtime state when GODEBUG=fips140=only is set. It confirms that FIPS mode is enforced.

```text
FIPS: chopconf.fips.enforced=true \
build.linked=true \
module.active=true \
runtime.enforced=true \
module=v1.0.0
```

--------------------------------

### Example: Incorrect ClickHouse Settings Create

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/clickhouse_config_errors_handling.md

Manifest for creating a new ClickHouseInstallation with invalid ClickHouse settings. This tests the operator's response to configuration errors that prevent ClickHouse from starting.

```yaml
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: incorrect-settings-create
spec:
  configuration:
    users:
      - name: default
        password: ""
        networks:
          - ipSourceRanges:
              - "0.0.0.0/0"
    clickhouse:
      configuration:
        users:
          - name: default
            password: ""
            networks:
              - ipSourceRanges:
                  - "0.0.0.0/0"
        profiles:
          - name: default
            settings:
              max_memory_usage: "-1" # Invalid value, should be a number
```

--------------------------------

### Create ZooKeeper Namespace (Quick Start)

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/zookeeper_setup.md

Creates a dedicated namespace for running ZooKeeper. This is an optional but recommended step for organizing resources.

```bash
kubectl create namespace zoo1ns
```

--------------------------------

### Download Operator Sources

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_build_from_sources.md

Use 'go get' to download the ClickHouse Operator source code from the repository. Ensure you have Go and its package manager installed.

```bash
go get github.com/altinity/clickhouse-operator
```

--------------------------------

### Install Grafana with Operator

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/grafana_setup.md

Execute this script to deploy Grafana, its service, and configure a recommended dashboard and Prometheus data source.

```bash
bash install-grafana-with-operator.sh
```

--------------------------------

### Deploy ZooKeeper (Quick Start)

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/zookeeper_setup.md

Applies the ZooKeeper deployment configuration to the specified namespace. Ensure the YAML file is present in your current directory.

```bash
kubectl apply -f zookeeper-1-node.yaml -n zoo1ns
```

--------------------------------

### Check ClickHouse Pod Status After Installation

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

After applying the ClickHouseInstallation, verify that the ClickHouse pods are running.

```bash
kubectl get pods -n test-clickhouse-operator
```

```text
NAME                    READY   STATUS    RESTARTS   AGE
chi-b3d29f-a242-0-0-0   1/1     Running   0          10m
```

--------------------------------

### Manual ClickHouse Operator Installation Script

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

This bash script provides a manual installation method for protected environments. It allows customization of namespaces and Docker images for the operator and metrics exporter.

```bash
#!/bin/bash

# Namespace to install operator into
OPERATOR_NAMESPACE="${OPERATOR_NAMESPACE:-test-clickhouse-operator}"
# Namespace to install metrics-exporter into
METRICS_EXPORTER_NAMESPACE="${OPERATOR_NAMESPACE}"
# Operator's docker image
OPERATOR_IMAGE="${OPERATOR_IMAGE:-altinity/clickhouse-operator:latest}"
# Metrics exporter's docker image
METRICS_EXPORTER_IMAGE="${METRICS_EXPORTER_IMAGE:-altinity/metrics-exporter:latest}"

# Setup clickhouse-operator into specified namespace
kubectl apply --namespace="${OPERATOR_NAMESPACE}" -f <( \
    curl -s https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-template.yaml | \
        OPERATOR_IMAGE="${OPERATOR_IMAGE}" \
        OPERATOR_NAMESPACE="${OPERATOR_NAMESPACE}" \
        METRICS_EXPORTER_IMAGE="${METRICS_EXPORTER_IMAGE}" \
        METRICS_EXPORTER_NAMESPACE="${METRICS_EXPORTER_NAMESPACE}" \
        envsubst \
)
```

--------------------------------

### Create ClickHouseInstallation Resource

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/custom_resource_explained.md

Defines a ClickHouseInstallation custom resource. Use `kubectl get clickhouseinstallations.clickhouse.altinity.com` to list resources.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse-installation-test"
```

--------------------------------

### Install Grafana Operator

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/grafana_setup.md

Run this script to set up the Grafana Operator CRDs, roles, service, and deployments.

```bash
bash install-grafana-operator.sh
```

--------------------------------

### Apply Settings with RollingUpdate

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_configuration.md

Example of setting spec.restart to 'RollingUpdate' to apply configuration changes and restart the server.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
...
spec:
  restart: "RollingUpdate"
...
```

--------------------------------

### Apply Initial ClickHouse Installation Manifest

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/chi_update_add_replication.md

Apply the initial manifest file for a simple stateless ClickHouse cluster. This command is used to set up the cluster before any updates.

```bash
kubectl -n dev apply -f 07-rolling-update-stateless-01-initial-position.yaml
```

--------------------------------

### Example Signed Git Commit Log

Source: https://github.com/altinity/clickhouse-operator/blob/master/CONTRIBUTING.md

This is an example of how your git log should appear after a signed commit. The 'Author' and 'Signed-off-by' lines must match.

```git
Author: John Doe <john.doe@example.com>
Date:   Mon Jan 24 12:34:56 2020 +0200

    Update README

    Signed-off-by: John Doe <john.doe@example.com>
```

--------------------------------

### Pod using PersistentVolumeClaim

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/storage.md

An example Pod manifest demonstrating the use of a `PersistentVolumeClaim` named `my-pvc` for its volume.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  volumes:
  - name: www
    persistentVolumeClaim:
      claimName: my-pvc
  containers:
  - name: nginx
    image: k8s.gcr.io/nginx-slim:0.8
    ports:
    - containerPort: 80
      name: web
    volumeMounts:
    - name: www
      mountPath: /usr/share/nginx/html
```

--------------------------------

### List ClickHouse Configuration Files

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/chi_update_add_replication.md

List the configuration files within the ClickHouse server's configuration directory inside a pod. This helps in understanding the current setup.

```bash
ls /etc/clickhouse-server/conf.d/
```

```bash
ls /etc/clickhouse-server/config.d
```

--------------------------------

### Simple Persistent Volume Configuration

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

Use this example when dynamic volume provisioning is available. It configures default data and log volume claim templates for a simple ClickHouse cluster.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "pv-simple"
spec:
  defaults:
    templates:
      dataVolumeClaimTemplate: data-volume-template
      logVolumeClaimTemplate: log-volume-template
  configuration:
    clusters:
      - name: "simple"
        layout:
          shardsCount: 1
          replicasCount: 1
  templates:
    volumeClaimTemplates:
      - name: data-volume-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
      - name: log-volume-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Mi
```

--------------------------------

### Apply ClickHouse Operator Installation Bundle

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_upgrade.md

Use this command to upgrade the ClickHouse Operator to the latest version by applying the installation bundle. This command should be run after ensuring CRDs are up-to-date.

```bash
kubectl apply -n kube-system -f https://github.com/Altinity/clickhouse-operator/raw/master/deploy/operator/clickhouse-operator-install-bundle.yaml
```

--------------------------------

### Start DevSpace Development Mode

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/devspace.md

Initiate DevSpace development mode, specifying the operator namespace and enabling Delve for remote debugging.

```bash
devspace dev --var=OPERATOR_NAMESPACE=kube-system --var=DEVSPACE_DEBUG=delve
```

--------------------------------

### Define ClickHouse Pod Template

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_configuration.md

Example of a ClickHouseInstallationTemplate defining a default pod template with a specific ClickHouse server image.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallationTemplate"

metadata:
  name: clickhouse-stable

spec:
  templates:
    podTemplates:
      - name: default
        spec:
          containers:
            - name: clickhouse-pod
              image: clickhouse/clickhouse-server:24.8
```

--------------------------------

### Example Kubernetes Manifest URL for Older Versions

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_installation_details.md

This URL format is used for applying operator manifests for versions prior to 0.16.0. Replace {OPERATOR_VERSION} with the specific version.

```bash
https://github.com/Altinity/clickhouse-operator/raw/{OPERATOR_VERSION}/deploy/operator/clickhouse-operator-install.yaml
```

--------------------------------

### StatefulSet with PersistentVolumeClaim Template

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/storage.md

A `StatefulSet` example that automatically provisions `PersistentVolumeClaim`s for each replica using `volumeClaimTemplates`. Each volume is named `www`.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
```

--------------------------------

### Install ClickHouse Operator via kubectl

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_installation_details.md

Use this command to apply the operator manifest directly from the GitHub repository. Ensure your kubectl is configured correctly.

```bash
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
```

--------------------------------

### Confirm TLS Setup with Strict Verification

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/security_hardening.md

Grep the operator logs for this message to confirm that strict TLS verification has been successfully registered and is active.

```text
TLS setup OK - root Cert registered (verify=Strict ...)

```

--------------------------------

### Generate StatefulSet with Custom Name Pattern

Source: https://github.com/altinity/clickhouse-operator/blob/master/release_notes.md

Example demonstrating how to use a custom pattern for generating StatefulSet names. This is useful for organizing and identifying ClickHouse resources.

```yaml
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: "example-chi"
  namespace: "chi-namespace"
spec:
  configuration:
    statefulSetNamePattern: "my-custom-name-{{ .Name }}-{{ .Shard }}"
```

--------------------------------

### Pod Volume Mount Example

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/storage.md

Defines how a container within a Pod mounts a volume. Ensure the volume name matches a definition in the Pod's `volumes` section.

```yaml
# ...
# excerpt from Pod or Pod Template manifest
# ...
containers:
  - name: myclickhouse
    image: clickhouse
    volumeMounts:
      - mountPath: "/var/lib/clickhouse"
        name: my-volume
```

--------------------------------

### Minimal PersistentVolumeClaim Example

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/storage.md

Defines a `PersistentVolumeClaim` requesting 1Gi of storage with `ReadWriteOnce` access mode. Omitting `storageClassName` uses the default.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
```

--------------------------------

### Secure ClickHouse Configuration with TLS

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/security_hardening.md

This example demonstrates a typical secure configuration for ClickHouse, enabling TLS/HTTPS and specifying ports. It includes the OpenSSL configuration for the server and placeholders for certificate files.

```yaml
spec:
  configuration:
    clusters:
    - name: default
      secure: "yes"
    settings:
      tcp_port: 9000 # keep for localhost
      tcp_port_secure: 9440
      https_port: 8443
    files:
      openssl.xml: |
        <clickhouse>
          <openSSL>
            <server>
              <certificateFile>/etc/clickhouse-server/config.d/server.crt</certificateFile>
              <privateKeyFile>/etc/clickhouse-server/config.d/server.key</privateKeyFile>
              <dhParamsFile>/etc/clickhouse-server/config.d/dhparam.pem</dhParamsFile>
              <verificationMode>none</verificationMode>
              <loadDefaultCAFile>true</loadDefaultCAFile>
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
            </server>
          </openSSL>
        </clickhouse>
      config.d/server.crt: |
        ***

      config.d/server.key: |
        ***

      config.d/dhparam.pem: |
        ***

```

--------------------------------

### Configure External Dictionary with Custom File

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/custom_resource_explained.md

An example of using .spec.configuration.files to define an external dictionary configuration file. Ensure the dictionaries_config setting in .spec.configuration.settings points to the correct file pattern.

```yaml
spec:
  configuration:
    settings:
      dictionaries_config: config.d/*.dict
    files:
      dict_one.dict: |
        <yandex>
          <dictionary>
        <name>one</name>
        <source>
            <clickhouse>
                <host>localhost</host>
                <port>9000</port>
                <user>default</user>
                <password/>
                <db>system</db>
                <table>one</table>
            </clickhouse>
        </source>
        <lifetime>60</lifetime>
        <layout><flat/></layout>
        <structure>
            <id>
                <name>dummy</name>
            </id>
            <attribute>
                <name>one</name>
                <expression>dummy</expression>
                <type>UInt8</type>
                <null_value>0</null_value>
            </attribute>
        </structure>
        </dictionary>
        </yandex>
```

--------------------------------

### Define Pod Template

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/requirements.md

Use the `podTemplates` array to define custom pod configurations. This example shows the basic structure for defining a pod template.

```yaml
spec:
  templates:
    podTemplates:
      - name: pod-template
        podDistribution:
```

--------------------------------

### Install ClickHouse Operator

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

Deploy the ClickHouse Operator into a specified namespace. This sets up necessary Kubernetes resources like Custom Resource Definitions, Service Accounts, and Deployments.

```text
namespace/test-clickhouse-operator created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com configured
serviceaccount/clickhouse-operator created
clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator configured
service/clickhouse-operator-metrics created
configmap/etc-clickhouse-operator-files created
configmap/etc-clickhouse-operator-confd-files created
configmap/etc-clickhouse-operator-configd-files created
configmap/etc-clickhouse-operator-templatesd-files created
configmap/etc-clickhouse-operator-usersd-files created
deployment.apps/clickhouse-operator created
```

--------------------------------

### Execute ClickHouse Operator Tests in Docker

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/README.md

Installs test dependencies, pulls the test runner image, and executes the test suite. Adjust COMPOSE_HTTP_TIMEOUT as needed for long-running tests.

```bash
pip3 install -U -r ./tests/image/requirements.txt
docker pull registry.gitlab.com/altinity-public/container-images/clickhouse-operator-test-runner:latest
COMPOSE_HTTP_TIMEOUT=1800 python3 ./tests/regression.py --only "/regression/e2e.test_operator/*"
```

--------------------------------

### Example: Incorrect ClickHouse Settings Update - Initial Position

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/clickhouse_config_errors_handling.md

Initial state manifest for updating an existing ClickHouseInstallation. This serves as the baseline before applying incorrect ClickHouse settings.

```yaml
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: incorrect-settings-update
spec:
  configuration:
    users:
      - name: default
        password: ""
        networks:
          - ipSourceRanges:
              - "0.0.0.0/0"
    clickhouse:
      configuration:
        users:
          - name: default
            password: ""
            networks:
              - ipSourceRanges:
                  - "0.0.0.0/0"
        profiles:
          - name: default
            settings:
              max_memory_usage: "1073741824"
```

--------------------------------

### Secure Communication Between ClickHouse Instances (Auto Secret)

Source: https://github.com/altinity/clickhouse-operator/blob/master/release_notes.md

Example of configuring secure communication between ClickHouse instances using an automatically generated secret. This is part of the secure cluster setup.

```yaml
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: secure-cluster-auto
spec:
  config:
    users:
      default:
        networks:
          - "0.0.0.0/0"
    clusters:
      - name: "cluster-1"
        secure: "true"
        internal_secure: "true"
        configuration:
          remote_servers:
            - id: "0"
              shard: 0
              replica: 0
              replica_name: "replica-0-0"
              host: "localhost"
              port: 9000
              user: "default"
              password: ""
              secure: "true"
              internal_secure: "true"

```

--------------------------------

### Build Manifests and Binaries, Push Changes

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/start_new_release.md

Build all manifests and binaries using the provided script, then commit and push the changes. This prepares the release artifacts.

```bash
./dev/go_build_all.sh
git add .
git commit -m "env: manifests"
git push
```

--------------------------------

### List ClickHouse Operator Deployments with Version Label

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_upgrade.md

Retrieve a list of ClickHouse Operator deployments, including their version, across all namespaces. This helps in quickly identifying the currently installed version.

```bash
kubectl get deployment --all-namespaces -L clickhouse.altinity.com/chop -l app=clickhouse-operator
```

--------------------------------

### Configure Minikube for Development

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/devspace.md

Configure Minikube to use the Docker driver, allocate CPU resources, set a Kubernetes version, start the cluster, and enable essential addons like ingress and metrics-server.

```bash
minikube config set driver docker
minikube config set cpus $(nproc)
minikube config set kubernetes-version 1.23.1
minikube start
minikube addons enable ingress
minikube addons enable ingress-dns
minikube addons enable metrics-server
```

--------------------------------

### Install ClickHouse Operator without Namespace Specification

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/quick_start.md

If no OPERATOR_NAMESPACE is specified, the installer script installs the operator into the kube-system namespace and watches custom resources in all namespaces.

```bash
cd ~
curl -s https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator-web-installer/clickhouse-operator-install.sh | bash
```

--------------------------------

### Check Upgrade Status and Verify Deployment

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_upgrade.md

After applying the installation bundle, use these commands to verify the upgrade status, check the new version, and inspect the deployment details to confirm the operator has been updated.

```bash
kubectl get deployment --all-namespaces -L clickhouse.altinity.com/chop -l app=clickhouse-operator
```

```bash
kubectl describe --all-namespaces deployment -l app=clickhouse-operator
```

--------------------------------

### Check Operator Installation

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/chi_update_add_replication.md

Verify that the clickhouse-operator is installed and running in the 'dev' namespace. This command lists all related resources.

```bash
kubectl -n dev get all,configmap
```

--------------------------------

### Configure ZooKeeper for Single Node Setup

Source: https://github.com/altinity/clickhouse-operator/blob/master/release_notes.md

Modify ZooKeeper manifests to use 'standaloneEnabled=false' for single-node setups, as recommended by ZooKeeper documentation for reconfigurations.

```yaml
standaloneEnabled: false
```

--------------------------------

### Build Metrics Exporter Wrapper Binary

Source: https://github.com/altinity/clickhouse-operator/blob/master/pkg/util/fips/acvp/README.md

Build the metrics-exporter wrapper binary with CGO_ENABLED=0 and the acvp_wrapper tag using a Dockerized Go environment. This command should be run from the repository root.

```bash
docker run --rm \
  -v "$PWD:/work" \
  -w /work \
  golang:1.26-alpine \
  sh -lc 'export PATH=$PATH:/usr/local/go/bin && CGO_ENABLED=0 go build -tags acvp_wrapper -o metrics-exporter ./cmd/metrics_exporter && ln -sf metrics-exporter metrics-exporter-acvp'
```

--------------------------------

### Example Kubernetes Secret for Operator Credentials

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/security_hardening.md

This YAML defines an example Kubernetes Secret of type Opaque, storing the 'clickhouse_operator' username and password.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-operator
type: Opaque
stringData:
  username: clickhouse_operator
  password: chpassword
```

--------------------------------

### Deploy Prometheus Instance

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/prometheus_setup.md

Applies a Prometheus instance configuration using a template, substituting the Prometheus namespace.

```bash
kubectl apply --namespace=prometheus --server-side -f <(wget -qO- https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/prometheus/prometheus-template.yaml | PROMETHEUS_NAMESPACE=prometheus envsubst)
```

--------------------------------

### Install ClickHouse Operator via Helm

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_installation_details.md

Add the Altinity Helm repository and install the ClickHouse Operator using the official Helm chart. This is available for versions 0.20.1 and later.

```bash
helm repo add clickhouse-operator https://helm.altinity.com
helm install clickhouse-operator clickhouse-operator/altinity-clickhouse-operator
```

--------------------------------

### Patch Persistent Volume to Remove ClaimRef

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/keeper_migration_from_23_to_24.md

After deleting the old CHK installation and PVC, use this command to remove the `claimRef` from the Persistent Volume (PV). This makes the volume available for remounting by the new CHK installation.

```bash
kubectl patch pv $PV -p '{"spec":{"claimRef": null}}'
```

--------------------------------

### Use ClickHouseInstallation Templates

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/requirements.md

Specify a list of `ClickHouseInstallationTemplate` resource names to merge with the current CHI manifest for rendering Kubernetes resources.

```yaml
spec:
  # List of templates used by a CHI
  useTemplates:
```

--------------------------------

### Verify Custom Resource Definitions

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_installation_details.md

Check if the Custom Resource Definitions for ClickHouse installations have been created.

```bash
kubectl get customresourcedefinitions
```

--------------------------------

### Build Operator Wrapper Binary

Source: https://github.com/altinity/clickhouse-operator/blob/master/pkg/util/fips/acvp/README.md

Build the operator wrapper binary with CGO_ENABLED=0 and the acvp_wrapper tag using a Dockerized Go environment. This command should be run from the repository root.

```bash
docker run --rm \
  -v "$PWD:/work" \
  -w /work \
  golang:1.26-alpine \
  sh -lc 'export PATH=$PATH:/usr/local/go/bin && CGO_ENABLED=0 go build -tags acvp_wrapper -o clickhouse-operator ./cmd/operator && ln -sf clickhouse-operator clickhouse-operator-acvp'
```

--------------------------------

### Get Grafana Service Details

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/grafana_setup.md

Retrieve information about the Grafana service to confirm its status and ClusterIP.

```bash
kubectl --namespace=grafana get service grafana-service
```

--------------------------------

### Configure Reconciling Cycle

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/requirements.md

Use the `.spec.reconciling` object to tune the reconciling cycle for a ClickHouse Installation (CHI).

```yaml
spec:
  reconciling:
```

--------------------------------

### Compare Image Manifest Byte-for-Byte

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/fips_evidence_verification.md

Compare the raw image manifest list byte-for-byte against the archived manifest file. An empty diff indicates a match.

```bash
docker buildx imagetools inspect --raw \
  altinity/clickhouse-operator:0.27.1 \
  | diff - release-evidence/clickhouse-operator__0.27.1.manifest.json
```

--------------------------------

### Switch Docker to Minikube Environment

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/devspace.md

Evaluate the output of `minikube docker-env` to configure your shell to use Minikube's Docker daemon.

```bash
eval $(minikube docker-env)
```

--------------------------------

### Configure Alertmanager Slack Integration

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/prometheus_setup.md

Sets up Alertmanager with Slack integration by applying a template after exporting necessary Slack and Alertmanager configuration variables.

```bash
export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXXX/YYYYY/ZZZZZ
export SLACK_CHANNEL="#alerts-channel-name"
export PROMETHEUS_NAMESPACE=prometheus
export ALERT_MANAGER_EXTERNAL_URL=https://your.external-domain.for-alertmanger/    
kubectl apply --namespace=prometheus -f <( 
    wget -qO- https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/prometheus/prometheus-alertmanager-template.yaml |
    envsubst 
  )
```

--------------------------------

### Reproduce ACVP-build Evidence Locally

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/fips_evidence_verification.md

Build the ClickHouse Operator binary locally with FIPS and ACVP tags, then run the ACVP test vectors using `pkg/util/fips/acvp/run.sh`. This process verifies the binary against NIST ACVP responder tests.

```bash
GOFIPS140=v1.0.0 go build -tags acvp_wrapper \
  -o dev/bin/clickhouse-operator ./cmd/operator
pkg/util/fips/acvp/run.sh dev/bin/clickhouse-operator
go version -m dev/bin/clickhouse-operator | grep GOFIPS140
```

--------------------------------

### Define ClickHouseKeeperInstallation Resource Kind

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/requirements.md

Defines the 'ClickHouseKeeperInstallation' custom resource kind. This is used to specify ClickHouse Keeper cluster installations.

```yaml
kind: "ClickHouseKeeperInstallation"
```

--------------------------------

### Basic ClickHouseInstallation with Keeper Reference

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/keeper_reference.md

This snippet shows a basic ClickHouseInstallation resource referencing a ClickHouseKeeperInstallation by name in its configuration.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: my-chi
spec:
  configuration:
    zookeeper:
      keeper:
        name: my-keeper
    clusters:
      - name: default
        layout:
          shardsCount: 1
          replicasCount: 2
```

--------------------------------

### Build Docker Image for ClickHouse Operator

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_build_from_sources.md

Create a Docker image for the ClickHouse Operator using the provided Dockerfile. This method does not require a Go compiler, but needs Docker and Kubernetes.

```bash
docker build -t altinity/clickhouse-operator -f ./dockerfile/operator/Dockerfile ./
```

--------------------------------

### Filter FIPSValidationFailed errors

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/security_hardening_fips.md

Use this command to filter for FIPSValidationFailed errors in the status of ClickHouse installations. This helps in diagnosing specific FIPS-related issues.

```bash
kubectl get chi -o json | jq -r '.items[].status.errors[]? | select(startswith("[FIPSValidationFailed]"))'
```

--------------------------------

### Verify ClickHouse Operator Pod Status

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/operator_installation_details.md

Check the status of the ClickHouse Operator pods in the 'kube-system' namespace to ensure it is running correctly after installation.

```bash
kubectl get pods --namespace kube-system
```

--------------------------------

### Propagate Annotations to Pods

Source: https://github.com/altinity/clickhouse-operator/blob/master/release_notes.md

Example demonstrating how to propagate annotations to ClickHouse pods. This is useful for applying custom labels or metadata to your ClickHouse instances.

```yaml
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: "example-chi"
spec:
  configuration:
    pods:
      annotations:
        "custom-annotation-key": "custom-annotation-value"
```

--------------------------------

### Get Metrics Exporter FIPS Info

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/fips.md

Retrieve FIPS module version and enabled status for the metrics-exporter. This output should indicate FIPS compliance.

```bash
metrics-exporter --fips-info
```

--------------------------------

### Get ClickHouse Operator FIPS Info

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/fips.md

Retrieve FIPS module version and enabled status for the clickhouse-operator. This output should indicate FIPS compliance.

```bash
clickhouse-operator --fips-info
```

--------------------------------

### Download Release Evidence Artifacts

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/fips_evidence_verification.md

Download the release evidence bundle for a specific tag using the GitHub CLI. This bundle contains digests, SBOMs, and build metadata.

```bash
gh run download --repo Altinity/clickhouse-operator \
  --name release-evidence-0.27.1 \
  --dir release-evidence/
```

--------------------------------

### Get Storage Classes

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/storage.md

Use `kubectl` to list available StorageClass objects in your Kubernetes cluster. This helps identify provisioners and default classes.

```bash
kubectl get storageclasses.storage.k8s.io
```

--------------------------------

### Verify Operator Security Configuration in Logs

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/security_hardening.md

Inspect operator logs at startup (INFO level) to confirm that the security configuration, including TLS settings, has been parsed and loaded correctly.

```bash
$ kubectl logs -n kube-system deploy/clickhouse-operator -c clickhouse-operator | grep -i security
... Config parsed ...
security:
  clickhouse:
    tls:
      verify: "Strict"
      minVersion: "1.3"
  ipc:
    mode: "Secure"
```

--------------------------------

### Configure ClickHouse Installation with ClickHouse Keeper

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/requirements.md

Configure a ClickHouseInstallation resource to connect to a clickhouse-keeper service. This involves specifying the zookeeper nodes and cluster layout.

```yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: clickhouse-with-clickhouse-keeper
spec:
  useTemplates:
    - name: clickhouse-version
  configuration:
    zookeeper:
      nodes:
        - host: clickhouse-keeper
          port: 2181
    clusters:
      - name: default
        layout:
          shardsCount: 2
          replicasCount: 1
```

--------------------------------

### ClickHouse Remote Servers Configuration for Secure Connections

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/security_hardening.md

Example of how the operator modifies the remote_servers configuration to use secure ports and flags for distributed queries.

```xml
<shard>
   <internal_replication>False</internal_replication>
      <replica>
          <host>***</host>
          <port>9440</port>
          <secure>1</secure>
      </replica>
   </shard>
```

--------------------------------

### Enable Operator Managed PV Provisioning

Source: https://github.com/altinity/clickhouse-operator/blob/master/release_notes.md

To enable operator-managed PV provisioning, use the specified configuration within your ClickHouseInstallation spec. This allows for volume rescaling without downtime.

```yaml
defaults:
  storageManagement:
    provisioner: Operator
```

--------------------------------

### List Zookeeper StatefulSets

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/zookeeper_setup.md

Use this command to list the statefulsets managing Zookeeper pods. This verifies the desired number of replicas are running and healthy.

```bash
kubectl get statefulset -n zoons
```

--------------------------------

### Define Files for Shard Replica

Source: https://github.com/altinity/clickhouse-operator/blob/master/tests/requirements/requirements.md

Use the `.spec.configuration.clusters[].layout.shards[].files` array to define the content of any settings file within a specific shard replica. These files are mounted in `/etc/clickhouse-server/config.d/`, `/etc/clickhouse-server/conf.d/`, or `/etc/clickhouse-server/users.d/`. Specified files override top-level and cluster-level file settings.

```yaml
spec:
  configuration:
    clusters:
      - name: cluster-name
        layout:
          shards:
            - name: shard-name
              replicas:
                - name: shard0-replica0
                  files:
```

--------------------------------

### ClickHouse Remote Servers Configuration

Source: https://github.com/altinity/clickhouse-operator/blob/master/docs/custom_resource_explained.md

Example of the ClickHouse configuration file's <remote_servers> section, corresponding to a cluster layout with multiple shards and replicas.

```xml
<yandex>
    <remote_servers>
        <all-counts>
        
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>192.168.1.1</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>192.168.1.2</host>
                    <port>9000</port>
                </replica>
            </shard>
            
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>192.168.1.3</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>192.168.1.4</host>
                    <port>9000</port>
                </replica>
            </shard>
            
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>192.168.1.5</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>192.168.1.6</host>
                    <port>9000</port>
                </replica>
            </shard>
            
        </all-counts>
    </remote_servers>
</yandex>
```