### Install project dependencies

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/02_docs_and_website.md

Install all required Python packages for the project by referencing the requirements.txt file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Start Docker Compose Services

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Use this command to start the necessary Docker Compose services for the playground environment.

```bash
docker compose --file docker/compose.yml up
```

--------------------------------

### Run StreamingRateExample with Gradle

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Executes the StreamingRateExample using Gradle. This example is useful for debugging streaming writes and micro-batch processing.

```bash
# From project root
CH_PORT=8123 ./gradlew :examples-scala-spark-4.0:runStreaming \
  -Dspark_binary_version=4.0 -Dscala_binary_version=2.13
```

--------------------------------

### Run StreamingRateExample with spark-submit

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Submits the StreamingRateExample using spark-submit. This example demonstrates micro-batch processing and continuous ingestion.

```bash
spark-submit \
  --class examples.StreamingRateExample \
  --master local[*] \
  --jars /path/to/clickhouse-spark-runtime-4.0_2.13-0.9.0-SNAPSHOT.jar \
  /path/to/examples-scala-spark-4.0.jar
```

--------------------------------

### Install and create a virtual environment with pyenv

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/02_docs_and_website.md

Install a specific Python version (3.9.13) and create a virtual environment named 'scc' using pyenv.

```bash
pyenv install 3.9.13
pyenv virtualenv 3.9.13 scc
```

--------------------------------

### Run SimpleBatchExample with Gradle

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Executes the SimpleBatchExample using Gradle. This example is good for debugging table creation, batch writes, and read operations.

```bash
# From project root
CH_PORT=8123 ./gradlew :examples-scala-spark-4.0:runBatch \
  -Dspark_binary_version=4.0 -Dscala_binary_version=2.13
```

--------------------------------

### Run ClickHouse Server with Docker

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Starts a ClickHouse server instance using Docker. Ensure ports 8123 and 9000 are accessible.

```bash
# Using Docker
docker run -d --name clickhouse-server \
  -p 8123:8123 -p 9000:9000 \
  --ulimit nofile=262144:262144 \
  clickhouse/clickhouse-server
```

--------------------------------

### Install pyenv and pyenv-virtualenv with Homebrew

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/02_docs_and_website.md

Use Homebrew to install pyenv and pyenv-virtualenv for managing Python versions and virtual environments.

```bash
brew install pyenv pyenv-virtualenv
```

--------------------------------

### Run All Tests

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/01_build_and_test.md

Execute all integration tests for the connector using this Gradle command. Ensure Docker and Docker Compose are installed and running.

```shell
./gradlew clean test
```

--------------------------------

### Conventional Commit Message Example

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/CONTRIBUTING.md

Follow the Conventional Commits specification for commit messages. This example shows a feature commit with a body and footer.

```markdown
feat(write): add support for batch insert optimization

This change improves write performance by batching inserts
when writing large datasets to ClickHouse.

Closes #123
```

--------------------------------

### Configure Spark Catalog Defaults

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/best_practices/01_deployment.md

Persist catalog configurations into `$SPARK_HOME/conf/spark-defaults.conf` to avoid using `--conf` flags when launching spark-shell or spark-sql. This example shows configurations for two different ClickHouse instances.

```properties
spark.sql.catalog.ck_01=com.clickhouse.spark.ClickHouseCatalog
spark.sql.catalog.ck_01.host=10.0.0.1
spark.sql.catalog.ck_01.protocol=http
spark.sql.catalog.ck_01.http_port=8123
spark.sql.catalog.ck_01.user=app
spark.sql.catalog.ck_01.password=pwd
spark.sql.catalog.ck_01.database=default

spark.sql.catalog.ck_02=com.clickhouse.spark.ClickHouseCatalog
spark.sql.catalog.ck_02.host=10.0.0.2
spark.sql.catalog.ck_02.protocol=http
spark.sql.catalog.ck_02.http_port=8123
spark.sql.catalog.ck_02.user=app
spark.sql.catalog.ck_02.password=pwd
spark.sql.catalog.ck_02.database=default
```

--------------------------------

### Configure ClickHouse Cluster Catalogs

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/configurations/01_catalog_configurations.md

Configure multiple ClickHouse instances in a cluster by giving each a unique catalog name. This example shows how to configure two instances, 'clickhouse1' and 'clickhouse2', using HTTPS.

```properties
spark.sql.catalog.clickhouse1                com.clickhouse.spark.ClickHouseCatalog
spark.sql.catalog.clickhouse1.host           10.0.0.1
spark.sql.catalog.clickhouse1.protocol       https
spark.sql.catalog.clickhouse1.http_port      8443
spark.sql.catalog.clickhouse1.user           default
spark.sql.catalog.clickhouse1.password
spark.sql.catalog.clickhouse1.database       default
spark.sql.catalog.clickhouse1.option.ssl     true

spark.sql.catalog.clickhouse2                com.clickhouse.spark.ClickHouseCatalog
spark.sql.catalog.clickhouse2.host           10.0.0.2
spark.sql.catalog.clickhouse2.protocol       https
spark.sql.catalog.clickhouse2.http_port      8443
spark.sql.catalog.clickhouse2.user           default
spark.sql.catalog.clickhouse2.password
spark.sql.catalog.clickhouse2.database       default
spark.sql.catalog.clickhouse2.option.ssl     true
```

--------------------------------

### Stop Docker Compose Services

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Stop the running Docker Compose services by pressing CTRL+C in the terminal where they were started.

```bash
CTRL+C
```

--------------------------------

### Serve the documentation website locally

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/02_docs_and_website.md

Use the mkdocs serve command to build and preview the documentation website locally. Access it via http://127.0.0.1:8000/.

```bash
mkdocs serve
```

--------------------------------

### Run StreamingRateExample with sbt

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Executes the StreamingRateExample using sbt. This is useful for testing continuous data ingestion and performance.

```bash
# From examples/scala/spark-4.0 directory
CH_PORT=8123 sbt "runMain examples.StreamingRateExample"
```

--------------------------------

### Connect to Kyuubi using Beeline

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Connect to the Kyuubi service using the beeline client. This is the recommended method for beginners.

```bash
docker exec -it kyuubi /opt/kyuubi/bin/beeline -u 'jdbc:kyuubi://0.0.0.0:10009/'
```

--------------------------------

### Run SimpleBatchExample with sbt

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Executes the SimpleBatchExample using sbt. The connector is automatically downloaded from Maven Central.

```bash
# From examples/scala/spark-4.0 directory
CH_PORT=8123 sbt "runMain examples.SimpleBatchExample"
```

--------------------------------

### Apply Code Formatting

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/CONTRIBUTING.md

Use the Spotless Gradle task to automatically format your code according to the project's style guidelines before committing.

```bash
./gradlew spotlessApply
```

--------------------------------

### Launch Spark SQL CLI with ClickHouse Catalog

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/02_play_with_spark_sql.md

Use this command to launch the Spark SQL CLI and configure it to use the ClickHouse catalog. Ensure the JAR paths are correct for your environment.

```shell
$SPARK_HOME/bin/spark-sql \
  --conf spark.sql.catalog.clickhouse=com.clickhouse.spark.ClickHouseCatalog \
  --conf spark.sql.catalog.clickhouse.host=${CLICKHOUSE_HOST:-127.0.0.1} \
  --conf spark.sql.catalog.clickhouse.protocol=http \
  --conf spark.sql.catalog.clickhouse.http_port=${CLICKHOUSE_HTTP_PORT:-8123} \
  --conf spark.sql.catalog.clickhouse.user=${CLICKHOUSE_USER:-default} \
  --conf spark.sql.catalog.clickhouse.password=${CLICKHOUSE_PASSWORD:-} \
  --conf spark.sql.catalog.clickhouse.database=default \
  --jars /path/clickhouse-spark-runtime-{{ spark_binary_version }}_{{ scala_binary_version }}:{{ stable_version }}.jar,/path/clickhouse-jdbc-{{ clickhouse_jdbc_version }}-all.jar
```

--------------------------------

### Run SimpleBatchExample with spark-submit

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Submits the SimpleBatchExample using spark-submit. Ensure the JAR paths are correct for your environment.

```bash
spark-submit \
  --class examples.SimpleBatchExample \
  --master local[*] \
  --jars /path/to/clickhouse-spark-runtime-4.0_2.13-0.9.0-SNAPSHOT.jar \
  /path/to/examples-scala-spark-4.0.jar
```

--------------------------------

### Build Docker Images for Development

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Build the Docker images for development, enabling snapshot versions of the Spark ClickHouse Connector.

```bash
DEV=1 docker/build-image.sh
```

--------------------------------

### Build Cross-Platform Docker Images with Buildx

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Optionally, use buildx to build cross-platform compatible Docker images.

```bash
BUILDX=1 docker/build-image.sh
```

--------------------------------

### Connect to Kyuubi using spark-sql

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Connect to the Kyuubi service using the spark-sql client.

```bash
docker exec -it kyuubi /opt/spark/bin/spark-sql
```

--------------------------------

### Connect to Kyuubi using DataGrip or DBeaver

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Configure a new Hive, Spark SQL, or Kyuubi datasource in DataGrip or DBeaver using the provided connection details.

```bash
jdbc:hive2://0.0.0.0:10009/
```

--------------------------------

### Clone the Repository

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/CONTRIBUTING.md

Clone your forked repository to your local machine and navigate into the project directory.

```bash
git clone https://github.com/YOUR_USERNAME/spark-clickhouse-connector.git
cd spark-clickhouse-connector
```

--------------------------------

### Publish to Maven Repository

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/03_private_release.md

Execute this Gradle command to publish the build artifacts to the configured Maven Repository.

```bash
./gradlew publish
```

--------------------------------

### Set Up Upstream Remote

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/CONTRIBUTING.md

Add the original repository as an upstream remote to fetch changes from the main project.

```bash
git remote add upstream https://github.com/ClickHouse/spark-clickhouse-connector.git
```

--------------------------------

### Launch Development Compose Services

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Launch the development Docker Compose services using a specific compose file and environment file.

```bash
docker compose --file docker/compose-dev.yml --env-file docker/.env-dev up
```

--------------------------------

### Alternative JAR Loading using Packages

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/02_play_with_spark_sql.md

This alternative to the --jars argument uses --packages to load the necessary ClickHouse connector JARs. This is useful to avoid manual JAR copying.

```shell
--repositories https://{maven-cental-mirror or private-nexus-repo} \
  --packages com.clickhouse.spark:clickhouse-spark-runtime-{{ spark_binary_version }}_{{ scala_binary_version }}:{{ stable_version }},com.clickhouse:clickhouse-jdbc:{{ clickhouse_jdbc_version }}:all
```

--------------------------------

### Clone the Connector Repository

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/01_build_and_test.md

Use this command to check out the source code from the GitHub repository.

```shell
git checkout https://github.com/ClickHouse/spark-clickhouse-connector.git
```

--------------------------------

### Inspect ClickHouse Table Structure

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Use DESCRIBE TABLE to view the schema of a ClickHouse table. Use SHOW CREATE TABLE to see the engine and settings.

```sql
-- Check table structure
DESCRIBE TABLE default.streaming_events;
```

```sql
-- Check table engine and settings
SHOW CREATE TABLE default.streaming_events;
```

--------------------------------

### Connect to Kyuubi using spark-shell

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Connect to the Kyuubi service using the spark-shell client.

```bash
docker exec -it kyuubi /opt/spark/bin/spark-shell
```

--------------------------------

### Launch Spark Shell with ClickHouse Catalog

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/03_play_with_spark_shell.md

Launch the Spark shell and configure it to use the ClickHouse catalog. This sets up the connection details for interacting with ClickHouse.

```shell
$SPARK_HOME/bin/spark-shell \
  --conf spark.sql.catalog.clickhouse=com.clickhouse.spark.ClickHouseCatalog \
  --conf spark.sql.catalog.clickhouse.host=${CLICKHOUSE_HOST:-127.0.0.1} \
  --conf spark.sql.catalog.clickhouse.protocol=http \
  --conf spark.sql.catalog.clickhouse.http_port=${CLICKHOUSE_HTTP_PORT:-8123} \
  --conf spark.sql.catalog.clickhouse.user=${CLICKHOUSE_USER:-default} \
  --conf spark.sql.catalog.clickhouse.password=${CLICKHOUSE_PASSWORD:-} \
  --conf spark.sql.catalog.clickhouse.database=default \
  --jars /path/clickhouse-spark-runtime-{{ spark_binary_version }}_{{ scala_binary_version }}:{{ stable_version }}.jar,/path/clickhouse-jdbc-{{ clickhouse_jdbc_version }}-all.jar
```

--------------------------------

### Run a Single Test Suite

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/01_build_and_test.md

To run a specific test suite, use the `--tests` flag with the Gradle command, specifying the suite name.

```shell
./gradlew test --tests=ConvertDistToLocalWriteSuite
```

--------------------------------

### Build Docker Images for Release

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Build the Docker images for the release version of the Spark ClickHouse Connector.

```bash
docker/build-image.sh
```

--------------------------------

### Configure pyenv in zshrc

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/02_docs_and_website.md

Add pyenv initialization commands to your ~/.zshrc file to enable pyenv and pyenv-virtualenv.

```bash
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```

--------------------------------

### Apply Spotless Formatting

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/AGENTS.md

Applies Spotless formatting to the code. This should be run for every Spark version touched. Specify Spark binary version using system property.

```bash
./gradlew spotlessApply -Dspark_binary_version=3.5
```

--------------------------------

### Build Local Connector JAR

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Builds a local connector JAR for debugging. This is optional and only needed if you are modifying connector code.

```bash
# From project root - build the local connector JAR
./gradlew :clickhouse-spark-runtime-4.0_2.13:shadowJar -Dspark_binary_version=4.0 -Dscala_binary_version=2.13
```

--------------------------------

### Run Tests with Specific Spark and Scala Versions

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/README.md

Run all integration tests, specifying the Spark and Scala binary versions. This is useful for testing compatibility.

```bash
./gradlew clean test -Dspark_binary_version=3.3 -Dscala_binary_version=2.13
```

--------------------------------

### Build Connector without Tests

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/01_build_and_test.md

Execute this Gradle command to build the connector JAR file, excluding test execution. The output JAR will be located in `spark-{{ spark_binary_version }}/clickhouse-spark-runtime/build/libs/`.

```shell
./gradlew clean build -x test
```

--------------------------------

### Configure PostgreSQL Catalog in Spark Defaults

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/internals/01_catalog.md

Configure a PostgreSQL database as a Spark catalog by adding these properties to `spark-defaults.conf`. This enables Spark to discover tables in the PostgreSQL instance.

```properties
# spark-defaults.conf
spark.sql.catalog.pg=org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
spark.sql.catalog.pg.url=jdbc:postgresql://<pg_host>:<pg_host>/<pg_db>
spark.sql.catalog.pg.driver=org.postgresql.Driver
spark.sql.catalog.pg.user=<pg_username>
spark.sql.catalog.pg.password=<pg_password>
```

--------------------------------

### Monitor Streaming Data in ClickHouse

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

SQL queries to monitor the data ingested by the StreamingRateExample in ClickHouse. Useful for verifying data integrity and performing basic analysis.

```sql
-- In ClickHouse client
SELECT count(*) FROM default.streaming_events;

SELECT 
  event_type, 
  count(*) as cnt, 
  avg(metric_value) as avg_metric
FROM default.streaming_events 
GROUP BY event_type;

SELECT 
  toStartOfMinute(event_time) as minute,
  count(*) as events_per_minute
FROM default.streaming_events 
GROUP BY minute
ORDER BY minute DESC
LIMIT 10;
```

--------------------------------

### Configure Nexus Repository Credentials for Gradle

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/03_private_release.md

Add these properties to `~/.gradle/gradle.properties` to authenticate with your company's private Nexus Repository for releases and snapshots.

```properties
mavenUser=xxx
mavenPassword=xxx
mavenReleasesRepo=xxx
mavenSnapshotsRepo=xxx
```

--------------------------------

### Localize virtual environment with pyenv

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/02_docs_and_website.md

Set the 'scc' virtual environment as the local environment for the current directory using pyenv.

```bash
pyenv local scc
```

--------------------------------

### Configure Single ClickHouse Instance Catalog

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/configurations/01_catalog_configurations.md

Use this configuration to register a single ClickHouse instance as a catalog named 'clickhouse'. Set the host, port, protocol, user, and database. Custom options for the ClickHouse client can also be specified.

```properties
####################################################################################
## register a catalog named "clickhouse"
####################################################################################
spark.sql.catalog.clickhouse                      com.clickhouse.spark.ClickHouseCatalog

####################################################################################
## basic configurations for "clickhouse" catalog
####################################################################################
spark.sql.catalog.clickhouse.host                 10.0.0.1
spark.sql.catalog.clickhouse.protocol             http
spark.sql.catalog.clickhouse.http_port            8123
spark.sql.catalog.clickhouse.user                 default
spark.sql.catalog.clickhouse.password
spark.sql.catalog.clickhouse.database             default

##############################################################################################
## custom options of clickhouse-client for "clickhouse" catalog
##############################################################################################
spark.sql.catalog.clickhouse.option.ssl                 false
spark.sql.catalog.clickhouse.option.async               false
spark.sql.catalog.clickhouse.option.client_name         spark
spark.sql.catalog.clickhouse.option.custom_http_params  async_insert=1,wait_for_async_insert=1
```

--------------------------------

### Spark SQL Operations with ClickHouse

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/03_play_with_spark_shell.md

Perform basic Spark SQL operations such as switching to the ClickHouse catalog, creating a database, showing databases, creating a table with specific properties, and writing data.

```scala
scala> spark.sql("use clickhouse")
res0: org.apache.spark.sql.DataFrame = []
```

```scala
scala> spark.sql("create database test_db")
res1: org.apache.spark.sql.DataFrame = []
```

```scala
scala> spark.sql("show databases").show
+--------+
|namespace|
+--------+
| default|
| system|
| test_db|
+--------+
```

```scala
scala> spark.sql("""
     | CREATE TABLE test_db.tbl (
     |   create_time TIMESTAMP NOT NULL,
     |   m           INT       NOT NULL COMMENT 'part key',
     |   id          BIGINT    NOT NULL COMMENT 'sort key',
     |   value       STRING
     | ) USING ClickHouse
     | PARTITIONED BY (m)
     | TBLPROPERTIES (
     |   engine = 'MergeTree()',
     |   order_by = 'id',
     |   settings.index_granularity = 8192
     | ) 
     | """)
res2: org.apache.spark.sql.DataFrame = []
```

```scala
// Entering paste mode (ctrl-D to finish)

spark.createDataFrame(Seq(
    ("2021-01-01 10:10:10", 1L, "1"),
    ("2022-02-02 10:10:10", 2L, "2")
)).toDF("create_time", "id", "value")
    .withColumn("create_time", to_timestamp($
```

```scala
create_time"))
    .withColumn("m", month($
```

```scala
create_time"))
    .select($
```

```scala
create_time, m, id, value)
    .writeTo("test_db.tbl")
    .append

// Exiting paste mode, now interpreting.
```

```scala
scala> spark.table("test_db.tbl").show
+-------------------+---+---+-----+
|        create_time|  m| id|value|
+-------------------+---+---+-----+
|2021-01-01 10:10:10|  1|  1|    1|
|2022-02-02 10:10:10|  2|  2|    2|
+-------------------+---+---+-----+
```

```scala
scala> spark.sql("DELETE FROM test_db.tbl WHERE id=1")
res3: org.apache.spark.sql.DataFrame = []
```

```scala
scala> spark.table("test_db.tbl").show
+-------------------+---+---+-----+
|        create_time|  m| id|value|
+-------------------+---+---+-----+
|2022-02-02 10:10:10|  2|  2|    2|
+-------------------+---+---+-----+
```

--------------------------------

### Build Spark ClickHouse Connector

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/AGENTS.md

Builds the Spark ClickHouse Connector with tests excluded. Specify Spark and Scala binary versions using system properties.

```bash
./gradlew clean build -x test -Dspark_binary_version=3.5 -Dscala_binary_version=2.13
```

--------------------------------

### Create a New Branch

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/CONTRIBUTING.md

Create a new branch for your feature or bug fix using a descriptive name. Use 'feature/' for new features and 'fix/' for bug fixes.

```bash
git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-description
```

--------------------------------

### Run Tests for Specific Spark and Scala Versions

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/CONTRIBUTING.md

Run tests targeting a specific Spark and Scala binary version by setting Gradle properties.

```bash
./gradlew clean test -Dspark_binary_version=3.5 -Dscala_binary_version=2.13
```

--------------------------------

### Spark SQL Operations with ClickHouse

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/02_play_with_spark_sql.md

Perform basic SQL operations like creating databases, tables, inserting data, and querying using the Spark SQL CLI connected to ClickHouse.

```sql
spark-sql> use clickhouse;
Time taken: 0.016 seconds
```

```sql
spark-sql> create database if not exists test_db;
Time taken: 0.022 seconds
```

```sql
spark-sql> show databases;
default
system
test_db
Time taken: 0.289 seconds, Fetched 3 row(s)
```

```sql
spark-sql> CREATE TABLE test_db.tbl_sql (
         >   create_time TIMESTAMP NOT NULL,
         >   m           INT       NOT NULL COMMENT 'part key',
         >   id          BIGINT    NOT NULL COMMENT 'sort key',
         >   value       STRING
         > ) USING ClickHouse
         > PARTITIONED BY (m)
         > TBLPROPERTIES (
         >   engine = 'MergeTree()',
         >   order_by = 'id',
         >   settings.index_granularity = 8192
         > );
Time taken: 0.242 seconds
```

```sql
spark-sql> insert into test_db.tbl_sql values
         > (timestamp'2021-01-01 10:10:10', 1, 1L, '1'),
         > (timestamp'2022-02-02 10:10:10', 2, 2L, '2')
         > as tabl(create_time, m, id, value);
Time taken: 0.276 seconds
```

```sql
spark-sql> select * from test_db.tbl_sql;
2021-01-01 10:10:10	1	1	1
2022-02-02 10:10:10	2	2	2
Time taken: 0.116 seconds, Fetched 2 row(s)
```

```sql
spark-sql> insert into test_db.tbl_sql select * from test_db.tbl_sql;
Time taken: 1.028 seconds
```

```sql
spark-sql> insert into test_db.tbl_sql select * from test_db.tbl_sql;
Time taken: 0.462 seconds
```

```sql
spark-sql> select count(*) from test_db.tbl_sql;
6
Time taken: 1.421 seconds, Fetched 1 row(s)
```

```sql
spark-sql> select * from test_db.tbl_sql;
2021-01-01 10:10:10	1	1	1
2021-01-01 10:10:10	1	1	1
2021-01-01 10:10:10	1	1	1
2022-02-02 10:10:10	2	2	2
2022-02-02 10:10:10	2	2	2
2022-02-02 10:10:10	2	2	2
Time taken: 0.123 seconds, Fetched 6 row(s)
```

```sql
spark-sql> delete from test_db.tbl_sql where id = 1;
Time taken: 0.129 seconds
```

```sql
spark-sql> select * from test_db.tbl_sql;
2022-02-02 10:10:10	2	2	2
2022-02-02 10:10:10	2	2	2
2022-02-02 10:10:10	2	2	2
Time taken: 0.101 seconds, Fetched 3 row(s)
```

--------------------------------

### Maven Repository for Snapshot Versions

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/01_get_the_library.md

Configure this repository in your Maven pom.xml if you intend to use SNAPSHOT versions of the ClickHouse Spark connector.

```xml
<repositories>
  <repository>
    <id>sonatype-oss-snapshots</id>
    <name>Sonatype OSS Snapshots Repository</name>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
  </repository>
</repositories>
```

--------------------------------

### Test Spark ClickHouse Connector

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/AGENTS.md

Runs tests for the Spark ClickHouse Connector. Specify Spark and Scala binary versions using system properties.

```bash
./gradlew test -Dspark_binary_version=3.5 -Dscala_binary_version=2.13
```

--------------------------------

### Create JDBC Table in Centralized Metastore

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/internals/01_catalog.md

Use this SQL command to register an external JDBC table in Spark's centralized metastore. This requires specifying the data source and connection options.

```sparksql
CREATE TABLE <db>.<tbl>
USING org.apache.spark.sql.jdbc
OPTIONS (
  url      "jdbc:mysql://<mysql_host>:<mysql_port>",
  dbtable  "<mysql_db>.<mysql_tbl>",
  user     "<mysql_username>",
  password "<mysql_password>"
);
```

--------------------------------

### Gradle Repository for Snapshot Versions

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/01_get_the_library.md

Include this repository configuration in your Gradle build if you need to use SNAPSHOT versions of the library.

```gradle
repositries {
  maven { url = "https://oss.sonatype.org/content/repositories/snapshots" }
}
```

--------------------------------

### Execute ClickHouse Native SQL via Spark

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/03_play_with_spark_shell.md

Execute native ClickHouse SQL commands directly from Spark using the `executeCommand` method. This is useful for operations not directly supported by Spark SQL.

```scala
scala> val options = Map(
     |     "host" -> "clickhouse",
     |     "protocol" -> "http",
     |     "http_port" -> "8123",
     |     "user" -> "default",
     |     "password" -> ""
     | )
```

```scala
scala> val sql = """
     | |CREATE TABLE test_db.person (
     | |  id    Int64,
     | |  name  String,
     | |  age Nullable(Int32)
     | |)
     | |ENGINE = MergeTree()
     | |ORDER BY id
     | """.stripMargin
```

```scala
scala> spark.executeCommand("com.clickhouse.spark.ClickHouseCommandRunner", sql, options)
```

```scala
scala> spark.sql("show tables in clickhouse_s1r1.test_db").show
+---------+---------+-----------+
|namespace|tableName|isTemporary|
+---------+---------+-----------+
|  test_db|   person|      false|
+---------+---------+-----------+
```

```scala
scala> spark.table("clickhouse_s1r1.test_db.person").printSchema
root
 |-- id: long (nullable = false)
 |-- name: string (nullable = false)
 |-- age: integer (nullable = true)
```

--------------------------------

### Query JDBC Table in Centralized Metastore

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/internals/01_catalog.md

Standard SQL queries for selecting from or inserting into a JDBC table registered in the centralized metastore.

```sparksql
SELECT * FROM <db>.<tbl>;
```

```sparksql
INSERT INTO <db>.<tbl> SELECT ...

```

--------------------------------

### Monitor ClickHouse Inserts

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Query the system.parts table to monitor the total rows and bytes inserted into specified tables.

```sql
-- Monitor inserts
SELECT 
  table,
  sum(rows) as total_rows,
  sum(bytes) as total_bytes
FROM system.parts
WHERE database = 'default' AND table IN ('streaming_events', 'employees')
GROUP BY table;
```

--------------------------------

### Authenticate as Admin using GraphQL API

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/conf/cloudbeaver-conf/README.md

Authenticates as an admin user using the CloudBeaver GraphQL API. This requires setting up a cookie jar to store session information.

```bash
curl 'http://0.0.0.0:8978/api/gql' \
  -X POST \
  -H 'content-type: application/json' \
  --cookie-jar /tmp/cookie.txt \
  --data '{ "query": "\n        query authLogin($provider: ID!, $credentials: Object!, $linkUser: Boolean) {\n            authToken: authLogin(\n                provider: $provider\n                credentials: $credentials\n                linkUser: $linkUser\n            ) {\n                authProvider\n            }\n        }\n    ", "variables": { "provider": "local", "credentials": { "user": "kyuubi", "password": "4E212BBF8F138808DB96B969716D1580" }, "linkUser": true } }'
```

--------------------------------

### Test with Custom ClickHouse Image

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/developers/01_build_and_test.md

Override the default ClickHouse Docker image used for integration tests by setting the `CLICKHOUSE_IMAGE` environment variable before running the Gradle test command.

```shell
CLICKHOUSE_IMAGE=custom-org/clickhouse-server:custom-tag ./gradlew test
```

--------------------------------

### View Recent ClickHouse Table Parts

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/examples/scala/spark-4.0/README.md

Examine recent data parts for a specific table, ordered by modification time, to understand data ingestion patterns.

```sql
-- Check recent parts
SELECT 
  partition,
  name,
  rows,
  bytes_on_disk,
  modification_time
FROM system.parts
WHERE database = 'default' AND table = 'streaming_events'
ORDER BY modification_time DESC
LIMIT 10;
```

--------------------------------

### Push Branch to Fork

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/CONTRIBUTING.md

Push your local feature branch to your forked repository on GitHub. This makes your changes available for creating a pull request.

```bash
git push origin feature/your-feature-name
```

--------------------------------

### Gradle Dependency for ClickHouse Spark Connector

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/01_get_the_library.md

Add this to your Gradle build file to include the ClickHouse Spark runtime and JDBC driver. Ensure you replace version placeholders with actual values.

```gradle
dependencies {
  implementation("com.clickhouse.spark:clickhouse-spark-runtime-{{ spark_binary_version }}_{{ scala_binary_version }}:{{ stable_version }}")
  implementation("com.clickhouse:clickhouse-jdbc:{{ clickhouse_jdbc_version }}:all") { transitive = false }
}
```

--------------------------------

### Generate Admin Password MD5

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/conf/cloudbeaver-conf/README.md

Generates an MD5 hash for the CloudBeaver admin password. Ensure the CB_ADMIN_PASSWORD environment variable is set.

```bash
CB_ADMIN_PASSWORD_MD5=`echo -n "$CB_ADMIN_PASSWORD" | md5sum | tr 'a-z' 'A-Z'`
CB_ADMIN_PASSWORD_MD5=${PASSWORD_MD5:0:32}
```

--------------------------------

### Query Table via PostgreSQL Catalog

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/internals/01_catalog.md

Access tables registered through the PostgreSQL catalog using Spark SQL. Tables are identified by the catalog name, database, and table name.

```sparksql
SELECT * FROM pg.<db>.<tbl>;
```

```sparksql
INSERT INTO pg.<db>.<tbl> SELECT ...

```

--------------------------------

### Maven Dependency for ClickHouse Spark Connector

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docs/quick_start/01_get_the_library.md

Add these dependencies to your Maven project's pom.xml to integrate the ClickHouse Spark runtime and JDBC driver. Customize version placeholders as needed.

```xml
<dependency>
  <groupId>com.clickhouse.spark</groupId>
  <artifactId>clickhouse-spark-runtime-{{ spark_binary_version }}_{{ scala_binary_version }}</artifactId>
  <version>{{ stable_version }}</version>
</dependency>
<dependency>
  <groupId>com.clickhouse</groupId>
  <artifactId>clickhouse-jdbc</artifactId>
  <classifier>all</classifier>
  <version>{{ clickhouse_jdbc_version }}</version>
  <exclusions>
    <exclusion>
      <groupId>*</groupId>
      <artifactId>*</artifactId>
    </exclusion>
  </exclusions>
</dependency>
```

--------------------------------

### Expose Connection to Anonymous User via GraphQL API

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/conf/cloudbeaver-conf/README.md

Grants anonymous user access to a specific connection using the CloudBeaver GraphQL API. This operation requires prior authentication and uses the stored session cookie.

```bash
curl 'http://0.0.0.0:8978/api/gql' \
  -X POST \
  -H 'content-type: application/json' \
  --cookie /tmp/cookie.txt \
  --data '{ "query": "\n        query setConnectionAccess($connectionId: ID!, $subjects: [ID!]!) {\n            setConnectionSubjectAccess(connectionId: $connectionId, subjects: $subjects) 
        }\n    ", "variables": { "connectionId": "kyuubi_hive-180f13452e0-749c09a3cdb63869", "subjects": ["user"] } }'
```

--------------------------------

### Remove Stopped Docker Compose Containers

Source: https://github.com/clickhouse/spark-clickhouse-connector/blob/main/docker/README.md

Remove the stopped containers created by Docker Compose.

```bash
docker compose --file docker/compose.yml rm
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.