### Install LZ4 Library

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/compression.md

Instructions for installing the LZ4 library on macOS using Homebrew and on Ubuntu/Debian systems using apt-get.

```bash
# macOS (Homebrew)
brew install lz4
```

```bash
# Ubuntu/Debian
apt-get install liblz4-dev
```

```bash
# Nix (if using flake.nix)
# Already included in development environment
```

--------------------------------

### Start ClickHouse Container

Source: https://github.com/couchemar/clickhouse_erl/blob/main/scripts/README.md

Commands to start the ClickHouse container using docker-compose and check its status.

```bash
# Start ClickHouse container
docker-compose up -d

# Check container status
docker ps | grep clickhouse
```

--------------------------------

### Install LZ4 on Ubuntu/Debian

Source: https://github.com/couchemar/clickhouse_erl/blob/main/c_src/README.md

Use this command to install the LZ4 compression library on Ubuntu or Debian-based systems.

```bash
sudo apt-get install liblz4-dev
```

--------------------------------

### Install LZ4 on Fedora/RHEL

Source: https://github.com/couchemar/clickhouse_erl/blob/main/c_src/README.md

Use this command to install the LZ4 compression library on Fedora or RHEL-based systems.

```bash
sudo dnf install lz4-devel
```

--------------------------------

### Handle Missing Compression Library

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/compression.md

Illustrates how to detect and handle cases where a required compression library is not installed.

```erlang
case clickhouse_erl:connect("localhost", 9000, #{compression => zstd}) of
    {error, {compression_library_missing, zstd}} ->
        io:format("ZSTD library not installed~n")
end.
```

--------------------------------

### Install LZ4 on macOS (Nix)

Source: https://github.com/couchemar/clickhouse_erl/blob/main/c_src/README.md

Use this command to install the LZ4 compression library on macOS using the Nix package manager.

```bash
nix-env -iA nixpkgs.lz4
```

--------------------------------

### Enable Query Profiling

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Example of enabling query profiling by sending progress headers and logging query details.

```erlang
% Enable query profiling
{ok, Result} = clickhouse_erl:query(
    Conn,
    <<"SELECT * FROM large_table">>,
    #{settings => #{
        <<"send_progress_in_http_headers">> => <<"1">>,
        <<"log_queries">> => <<"1">>,
        <<"log_query_threads">> => <<"1">
    }}
).
```

--------------------------------

### Control Output Format

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Example of configuring output formats for native JSON and date-time representation.

```erlang
% Control output format
{ok, Result} = clickhouse_erl:query(
    Conn,
    <<"SELECT * FROM users">>,
    #{settings => #{
        <<"output_format_native_json_as_string">> => <<"1">>,
        <<"date_time_output_format">> => <<"iso">
    }}
).
```

--------------------------------

### Verify LZ4 Installation

Source: https://github.com/couchemar/clickhouse_erl/blob/main/c_src/README.md

Verify that the LZ4 library is installed and accessible by checking its version using pkg-config.

```bash
pkg-config --modversion liblz4
```

--------------------------------

### Install LZ4 on macOS (Homebrew)

Source: https://github.com/couchemar/clickhouse_erl/blob/main/c_src/README.md

Use this command to install the LZ4 compression library on macOS using the Homebrew package manager.

```bash
brew install lz4
```

--------------------------------

### Start ClickHouse Erlang Application

Source: https://github.com/couchemar/clickhouse_erl/blob/main/README.md

Start the clickhouse_erl application before using its functions.

```erlang
application:start(clickhouse_erl).
```

--------------------------------

### EUnit Unit Test Example

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Demonstrates the structure of unit tests using EUnit, including testing specific inputs, error conditions, and connection lifecycles. Requires EUnit include.

```erlang
-module(clickhouse_erl_compression_tests).
-include_lib("eunit/include/eunit.hrl").

%% Test naming: <function>_<condition>_test()
encode_valid_input_test() ->
    Input = <<"test data">>,
    ?assertMatch({ok, _}, clickhouse_erl_compression:compress(Input, #{method => lz4})).

decode_empty_binary_test() ->
    ?assertEqual({error, invalid_format}, 
                 clickhouse_erl_compression:decompress(<<>>)).

%% Setup/cleanup for stateful tests
connection_lifecycle_test_() ->
    {setup,
     fun() -> 
         {ok, Conn} = test_helpers:connect(),
         Conn
     end,
     fun(Conn) -> clickhouse_erl:disconnect(Conn) end,
     fun(Conn) ->
         [?_assertMatch({ok, _}, clickhouse_erl:query(Conn, <<"SELECT 1">>)),
          ?_assertEqual(ok, clickhouse_erl:disconnect(Conn))]
     end}.

```

--------------------------------

### Unit Test with EUnit Fixtures for Setup/Teardown

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Demonstrates setup and teardown for unit tests using EUnit fixtures. It establishes a connection before tests run and disconnects afterward, asserting query results and disconnection status.

```erlang
connection_test_() ->
    {setup,
     fun() ->
         % Setup
         {ok, Conn} = test_helpers:connect(),
         Conn
     end,
     fun(Conn) ->
         % Teardown
         clickhouse_erl:disconnect(Conn)
     end,
     fun(Conn) ->
         % Tests
         [?_assertMatch({ok, _}, clickhouse_erl:query(Conn, <<"SELECT 1">>)),
          ?_assertEqual(ok, clickhouse_erl:disconnect(Conn))]
     end}.
```

--------------------------------

### Erlang Test Helpers Module

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Centralize common test setup, configuration, and utilities. Add when the same setup code appears in 3+ test files.

```erlang
-module(test_helpers).
-export([connect/0, disconnect/1, test_config/0]).

connect() ->
    Config = test_config(),
    Host = maps:get(host, Config),
    Port = maps:get(port, Config),
    clickhouse_erl:connect(Host, Port, Config).

test_config() ->
    #{"host" => "localhost", 
      "port" => 9000, 
      "user" => "default",
      "database" => "default"}.

disconnect(Conn) ->
    clickhouse_erl:disconnect(Conn).
```

--------------------------------

### Setting Resource Limits in ClickHouse Erlang

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Provides an example of setting resource limits for production queries using binary strings for keys and values.

```erlang
#{} => #{settings => #{
       <<"max_execution_time">> => <<"60">>,
       <<"max_memory_usage">> => <<"10000000000">>,
       <<"max_rows_to_read">> => <<"1000000000">>
   }}
```

--------------------------------

### Backward Compatible Queries

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_parameters.md

These examples show how to execute queries without parameters, maintaining backward compatibility. They also demonstrate equivalent calls using empty parameter maps or lists.

```erlang
% Query without parameters option (backward compatible)
{ok, Result} = clickhouse_erl:query(Conn, <<"SELECT 1">>).
```

```erlang
% Query with empty parameters map (equivalent to no parameters)
{ok, Result} = clickhouse_erl:query(Conn, <<"SELECT 1">>, #{}).
```

```erlang
% Query with empty parameters list (sends empty Parameters field)
{ok, Result} = clickhouse_erl:query(Conn, <<"SELECT 1">>, #{parameters => []}).
```

--------------------------------

### Start tcpdump Capture

Source: https://github.com/couchemar/clickhouse_erl/blob/main/scripts/README.md

Starts packet capture on port 9000 within the ClickHouse container. Supports custom filenames.

```bash
# Start capture with auto-generated filename
./scripts/start_tcpdump.sh

# Start capture with custom filename
./scripts/start_tcpdump.sh my_test_capture.pcap
```

--------------------------------

### Limit Query Execution Resources

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Example of setting query execution time, memory usage, and thread limits.

```erlang
% Limit query execution resources
{ok, Result} = clickhouse_erl:query(
    Conn,
    <<"SELECT count(*) FROM events WHERE date >= today() - 7">>,
    #{settings => #{
        <<"max_execution_time">> => <<"30">>,        % 30 seconds timeout
        <<"max_memory_usage">> => <<"5000000000">>,  % 5GB memory limit
        <<"max_threads">> => <<"8">>                 % Use 8 threads
    }}
).
```

--------------------------------

### Run Integration Tests with Docker Compose

Source: https://github.com/couchemar/clickhouse_erl/blob/main/README.md

Start a ClickHouse server using Docker Compose, run unit tests, and then shut down the server. This is for testing against a live database instance.

```bash
docker-compose up -d
rebar3 eunit
docker-compose down
```

--------------------------------

### Erlang Test Helper Import Pattern

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Example of how to import functions from the test_helpers module.

```erlang
-import(test_helpers, [connect/0, disconnect/1]).
```

--------------------------------

### Erlang Generator Import Pattern

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Example of how to import generators from the generators module.

```erlang
-import(generators, [binary_string_gen/0, compression_method_gen/0]).
```

--------------------------------

### Log Entry Format Example

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/server_logs.md

Illustrates the structure of a log entry map received by the `on_log` callback. Each entry contains details like event time, hostname, query ID, thread ID, priority, source module, and the log message text.

```erlang
#{
    <<"event_time">> => 1700000000,           %% Unix timestamp
    <<"event_time_microseconds">> => 123456,  %% Microseconds component
    <<"host_name">> => <<"server1">>,         %% Server hostname
    <<"query_id">> => <<"abc-123">>,          %% Query ID
    <<"thread_id">> => 42,                    %% Thread ID
    <<"priority">> => 6,                      %% Log priority (1=Fatal..8=Trace)
    <<"source">> => <<"executeQuery">>,       %% Source module
    <<"text">> => <<"Read 100 rows">>         %% Log message
}
```

--------------------------------

### PropEr Property Test Example

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Illustrates property-based testing with PropEr, focusing on invariants and pure functions without external dependencies. Does not use EUnit include or test wrappers.

```erlang
-module(prop_clickhouse_erl_compression).
-include_lib("proper/include/proper.hrl").

%% No EUnit include
%% No test wrappers

%% Property naming: prop_<invariant_name>()
prop_encode_decode_roundtrip() ->
    ?FORALL(Data, binary(),
        begin
            {ok, Encoded} = clickhouse_erl_compression:compress(Data, #{method => lz4}),
            {ok, Decompressed, _} = clickhouse_erl_compression:decompress(Encoded),
            Data =:= Decompressed
        end).

prop_compression_preserves_size() ->
    ?FORALL({Data, Method}, {binary(), compression_method()},
        begin
            {ok, Compressed} = clickhouse_erl_compression:compress(Data, #{method => Method}),
            {ok, Decompressed, _} = clickhouse_erl_compression:decompress(Compressed),
            byte_size(Data) =:= byte_size(Decompressed)
        end).

%% Generators
compression_method() ->
    oneof([lz4, zstd, none]).

```

--------------------------------

### on_log Callback with INSERT

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/server_logs.md

Illustrates that the `on_log` callback can be used with the `insert/4` function, similar to `query/3`. This example shows logging the text of log entries during an insert operation.

```erlang
{ok, _} = clickhouse_erl:insert(Conn, SQL, Input, #{
    settings => [{<<"send_logs_level">>, <<"information">>}],
    on_log => fun(Entry) ->
        io:format("INSERT log: ~s~n", [maps:get(<<"text">>, Entry)]),
        ok
    end
}).
```

--------------------------------

### Handle Connection Failures Gracefully (Erlang)

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Implement robust error handling for database connections. This example demonstrates skipping tests if a connection cannot be established, ensuring test stability.

```erlang
prop_my_property() ->
    ?FORALL(Input, input_gen(),
        begin
            case test_helpers:connect() of
                {ok, Conn} ->
                    try
                        % Test logic
                        test_query(Conn, Input)
                    after
                        clickhouse_erl:disconnect(Conn)
                    end;
                {error, _Reason} ->
                    true  % Skip if connection fails
            end
        end).

```

--------------------------------

### Erlang Integration Property Test Configuration

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Example of configuring PropEr for integration tests with reduced iteration counts. Pass {numtests, 50} to proper:quickcheck/2 to prevent timeouts.

```erlang
prop_my_test() ->
    Result = proper:quickcheck(
        prop_my_property(),
        [{numtests, 50}, {to_file, user}, {max_size, 20}]
    ),
    ?assert(Result =:= true).
```

--------------------------------

### Compile Project with LZ4 NIF

Source: https://github.com/couchemar/clickhouse_erl/blob/main/c_src/README.md

Compile the project using rebar3 after the LZ4 dependency has been successfully installed. This will automatically build the NIF.

```bash
rebar3 compile
```

--------------------------------

### Nullable Data Type Examples

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Demonstrates how to create a table with nullable columns, insert data including null values, and query nullable data. Nullable fields are represented as {null} during insertion and can be {value, ...} or {null} during retrieval.

```erlang
CreateSQL = <<"
    CREATE TABLE IF NOT EXISTS users (
        id UInt64,
        name String,
        email Nullable(String),
        age Nullable(Int64),
        tags Nullable(Array(String))
    ) ENGINE = MergeTree()
    ORDER BY id
">>,
{ok, _} = clickhouse_erl:query(Conn, CreateSQL),

% Insert nullable data
SQL = <<"INSERT INTO users (id, name, email, age, tags) VALUES">>,
Input = [
    #{name => <<
```

--------------------------------

### Insert Multiple Rows

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/insert_quick_reference.md

Example for inserting multiple rows. The 'data' list for each column should contain the same number of elements, corresponding to the rows being inserted.

```erlang
Input = [
    #{name => <<"id">>, type => <<"UInt64">>, data => [1, 2, 3]},
    #{name => <<"name">>, type => <<"String">>, data => [<<"Alice">>, <<"Bob">>, <<"Charlie">>]} 
],
{ok, _} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Erlang Representation of ClickHouse Arrays

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

Demonstrates how ClickHouse Array types are represented as Erlang lists, including examples for integers, strings, and empty arrays.

```erlang
% Array(Int64)
[1, 2, 3, 4, 5]

% Array(String)
[<<"tag1">>, <<"tag2">>, <<"tag3">>]

% Empty array
[]
```

--------------------------------

### Example: Test Binary String Encoding

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/GENERATOR_USAGE.md

Tests the encoding and decoding of UTF-8 strings using `binary_string_gen`. Ensures that encoded strings can be correctly decoded back to their original form.

```erlang
prop_string_encoding() ->
    ?FORALL(Str, binary_string_gen(),
        begin
            Encoded = encode_string(Str),
            {ok, Decoded, <<>>} = decode_string(Encoded),
            Str =:= Decoded
        end).
```

--------------------------------

### Handle Compression Errors Gracefully

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/compression.md

Implement error handling for cases where the compression library is missing. This example provides a fallback to an uncompressed connection if LZ4 or ZSTD libraries are unavailable.

```erlang
connect_with_compression(Host, Port, Method) ->
    case clickhouse_erl:connect(Host, Port, #{compression => Method}) of
        {ok, Conn} ->
            {ok, Conn};
        {error, {compression_library_missing, _}} ->
            % Fallback to no compression
            io:format("Compression library missing, using uncompressed connection~n"),
            clickhouse_erl:connect(Host, Port);
        {error, Reason} ->
            {error, Reason}
    end.
```

--------------------------------

### Document Property Requirements (Erlang)

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Document the requirements and validation aspects of a property test using comments. This example specifies that the compression round-trip consistency test validates multiple requirements.

```erlang
%% Property 1: Compression Round-Trip Consistency
%% Validates: Requirements 3.1, 3.2, 3.3, 3.4
prop_compression_roundtrip_consistency() ->
    ?FORALL({Data, Method}, {binary_data_gen(), compression_method_gen()},
        begin
            {ok, Compressed} = compress(Data, Method),
            {ok, Decompressed, _} = decompress(Compressed),
            Data =:= Decompressed
        end).

```

--------------------------------

### Example: Test Float64 Encoding

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/GENERATOR_USAGE.md

Tests the encoding and decoding of 64-bit floating-point numbers using `float64_gen`. Handles special cases like NaN to ensure accurate representation.

```erlang
prop_float64_encoding() ->
    ?FORALL(F, float64_gen(),
        begin
            Encoded = encode_float64(F),
            {ok, Decoded, <<>>} = decode_float64(Encoded),
            case F of
                nan -> Decoded =:= nan;
                _ -> F =:= Decoded
            end
        end).
```

--------------------------------

### Setting up ETS Table with Sample Data

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/streaming-insert.md

Prepares an ETS table with sample data for streaming. This includes creating the table, populating it with records, and defining the ClickHouse column schema.

```erlang
%% Create ETS table with sample data
ets:new(my_data, [named_table, ordered_set, public]),
lists:foreach(fun(Id) ->
    Name = iolist_to_binary(io_lib:format("user_~4..0B", [Id])),
    Score = rand:uniform(10000) / 100.0,
    ets:insert(my_data, {Id, Name, Score})
end, lists:seq(1, 10000)).

%% Column definitions (schema only — no data key needed)
Columns = [
    #{name => <<"id">>, type => <<
```

```erlang
<<
```

```erlang
UInt32">>},
    #{name => <<"name">>, type => <<
```

```erlang
String">>},
    #{name => <<"score">>, type => <<
```

```erlang
Float64">>}
].

SQL = <<"INSERT INTO users (id, name, score) VALUES">>.
```

--------------------------------

### Connect to ClickHouse with Defaults

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Establishes a basic connection to ClickHouse using default parameters and demonstrates executing a simple query.

```erlang
% Connect with defaults
{ok, Conn} = clickhouse_erl:connect("localhost", 9000),

% Use the connection
{ok, Result} = clickhouse_erl:query(Conn, <<"SELECT 1">>),

% Clean up
ok = clickhouse_erl:disconnect(Conn).
```

--------------------------------

### Migrating Query Settings to Keyword List Format

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Demonstrates an alternative migration path from the verbose protocol format to the keyword list format for query settings. This offers another way to simplify query option specification.

```erlang
% Or keyword list format (alternative)
clickhouse_erl:query(Conn, SQL, #{
    settings => [
        {<<"max_threads">>, <<"4">>},
        {<<"max_memory_usage">>, <<"10000000000">>}
    ]
}).
```

--------------------------------

### LowCardinality Data Representation

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

This example shows that the data representation remains the same when migrating to LowCardinality types.

```erlang
% No code changes needed - values are the same
{1, <"click">, <"button">}
```

--------------------------------

### Create Table with Decimal Types

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/extended_types.md

Demonstrates creating a table with Decimal64 and Decimal32 columns. Ensure the scale and precision limits are respected.

```erlang
clickhouse_erl:query(Conn, <<"
    CREATE TABLE financial_data (
        id UInt32,
        price Decimal64(18, 4),
        quantity Decimal32(9, 2)
    ) ENGINE = Memory
>>).
```

--------------------------------

### Handle Server-Side Compression Errors

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/compression.md

Provides examples for handling various server-side compression-related errors during query execution.

```erlang
case clickhouse_erl:query(Conn, <<"SELECT * FROM table">>) of
    {error, {server_error, unknown_compression_method}} ->
        io:format("Server doesn't support this compression method~n");
    {error, {server_error, cannot_decompress}} ->
        io:format("Server failed to decompress data~n");
    {error, {server_error, cannot_compress}} ->
        io:format("Server failed to compress response~n")
end.
```

--------------------------------

### Erlang Representation of Nested ClickHouse Tuples

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

Examples of nested Tuple structures in Erlang, illustrating how tuples can contain other tuples.

```erlang
% Tuple(Tuple(Int64, Int64), String)
{{100, 200}, <<"description">>}

% Tuple(name String, address Tuple(city String, zip Int32))
{<<"Alice">>, {<<"San Francisco">>, 94102}}
```

--------------------------------

### Migrating Query Settings to Simple Map Format

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Shows the transition from the verbose protocol format to the recommended simple map format for query settings. This simplifies the structure for specifying query options.

```erlang
% Before (verbose protocol format)
clickhouse_erl:query(Conn, SQL, #{
    settings => [
        #{key => <<"max_threads">>, value => <<"4">>, 
          important => false, custom => false, obsolete => false},
        #{key => <<"max_memory_usage">>, value => <<"10000000000">>, 
          important => false, custom => false, obsolete => false}
    ]
}).

% After (simple map format - recommended)
clickhouse_erl:query(Conn, SQL, #{
    settings => #{
        <<"max_threads">> => <<"4">>,
        <<"max_memory_usage">> => <<"10000000000">>
    }
}).
```

--------------------------------

### Streaming Inserts with Different Compression Settings

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/streaming-insert.md

Demonstrates how to configure compression for streaming inserts. Supported options include LZ4, ZSTD, and disabling compression.

```erlang
%% With LZ4 compression
{ok, Result} = clickhouse_erl:streaming_insert(Conn, SQL, Opts, #{compression => lz4}).
```

```erlang
%% With ZSTD compression
{ok, Result} = clickhouse_erl:streaming_insert(Conn, SQL, Opts, #{compression => zstd}).
```

```erlang
%% Compression disabled
{ok, Result} = clickhouse_erl:streaming_insert(Conn, SQL, Opts, #{compression => disabled}).
```

--------------------------------

### Run Unit Tests with Rebar3

Source: https://github.com/couchemar/clickhouse_erl/blob/main/README.md

Execute unit tests using the rebar3 build tool. This is a standard command for running EUnit tests.

```bash
# Unit tests
rebar3 eunit
```

--------------------------------

### Deeply Nested Composite Type Example

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

Demonstrates a deeply nested composite type structure, illustrating the 5-level nesting limit.

```erlang
% 5 levels: Array -> Nullable -> Tuple -> Array -> String
% Array(Nullable(Tuple(String, Array(String))))
[
    {value, {<"name">, [<"tag1">, <"tag2">]}},
    {null},
    {value, {<"other">, []}}
]
```

--------------------------------

### Execute Query with Timeout Option

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Execute a query with a specified timeout. This example shows how to handle both successful query completion and timeout errors.

```erlang
SQL = <<"SELECT * FROM large_table">>,
Options = #{timeout => 60000},  % 60 seconds

case clickhouse_erl:query(Conn, SQL, Options) of
    {ok, Result} ->
        io:format("Query completed~n");
    {error, {timeout_error, _}} ->
        io:format("Query timed out after 60 seconds~n")
end.
```

--------------------------------

### Running Integration Property Tests

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Commands to execute integration property tests using rebar3. Use `rebar3 eunit` for all tests or specify a module for targeted execution.

```bash
# Integration property tests run with unit tests
rebar3 eunit

# Specific module
rebar3 eunit --module=clickhouse_erl_connection_backward_compatibility_property_tests
```

--------------------------------

### Query with Simple Map Settings

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Pass query settings using the recommended map format. Keys and values must be binary strings. Default flags are applied.

```erlang
{ok, Conn} = clickhouse_erl:connect("localhost", 9000).

{ok, Result} = clickhouse_erl:query(
    Conn,
    <<"SELECT * FROM large_table">>,
    #{settings => #{
        <<"max_threads">> => <<"4">>,
        <<"max_memory_usage">> => <<"10000000000">
    }}
).
```

--------------------------------

### Insert Single Row

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/insert_quick_reference.md

Example of inserting a single row into a table. Ensure the 'data' list for each column contains exactly one value.

```erlang
Input = [
    #{name => <<"id">>, type => <<"UInt64">>, data => [1]},
    #{name => <<"name">>, type => <<"String">>, data => [<<"Alice">>]} 
],
{ok, _} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Erlang Maps with Complex Value Types

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

Provides examples of Erlang maps where the values are complex types, such as Arrays of Int64 or Tuples of String and Int64.

```erlang
% Map(String, Array(Int64))
#{<<"a">> => [1, 2, 3], <<"b">> => [4, 5]}

% Map(String, Tuple(String, Int64))
#{<<"user1">> => {<<"Alice">>, 25}, <<"user2">> => {<<"Bob">>, 30}}
```

--------------------------------

### Query with Keyword List Settings

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Pass query settings using the keyword list format, where each setting is a {Key, Value} tuple. Keys and values must be binary strings.

```erlang
{ok, Result} = clickhouse_erl:query(
    Conn,
    <<"SELECT * FROM large_table">>,
    #{settings => [
        {<<"max_threads">>, <<"4">>},
        {<<"max_memory_usage">>, <<"10000000000">>}
    ]}
).
```

--------------------------------

### Erlang Representation of Arrays of Tuples

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

Shows how ClickHouse Arrays containing Tuples are represented in Erlang, using a specific example of String and Int64 tuples.

```erlang
% Array(Tuple(String, Int64))
[{<|"Alice">|>, 25}, {<|"Bob">|>, 30}, {<|"Charlie">|>, 35}]
```

--------------------------------

### Using Binary Strings for ClickHouse Erlang Settings

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_settings.md

Illustrates the correct usage of binary strings for keys and values in ClickHouse Erlang query settings, contrasting it with incorrect usage of atoms or strings.

```erlang
% ✅ Good: Binary strings
#{} => #{settings => #{<<"max_threads">> => <<"4">>}}

% ❌ Bad: Atoms or strings
#{} => #{settings => #{max_threads => "4"}}
```

--------------------------------

### Insert and Query Tuple Data in ClickHouse Erlang

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Demonstrates how to create a table with tuple columns, insert data into these tuple columns, and then query the data using clickhouse_erl.

```erlang
% Create table with tuples
CreateSQL = <<"\n    CREATE TABLE IF NOT EXISTS locations (\n        id UInt64,\n        name String,\n        coordinates Tuple(Float64, Float64),\n        metadata Tuple(city String, country String, population Int64)\n    ) ENGINE = MergeTree()\n    ORDER BY id\n">>,
{ok, _} = clickhouse_erl:query(Conn, CreateSQL),

% Insert tuple data
SQL = <<"INSERT INTO locations (id, name, coordinates, metadata) VALUES">>,
Input = [
    #{name => <<
```

--------------------------------

### INSERT with All Data Types using ClickHouse Erlang

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Shows how to insert data covering all supported ClickHouse data types into a 'type_examples' table. This includes various integer types, floats, strings, booleans, dates, and datetimes with different precision.

```erlang
CreateSQL = <<"
    CREATE TABLE IF NOT EXISTS type_examples (
        col_uint8 UInt8,
        col_uint16 UInt16,
        col_uint32 UInt32,
        col_uint64 UInt64,
        col_int8 Int8,
        col_int16 Int16,
        col_int32 Int32,
        col_int64 Int64,
        col_float32 Float32,
        col_float64 Float64,
        col_string String,
        col_bool Bool,
        col_date Date,
        col_datetime DateTime,
        col_datetime64 DateTime64(6)
    ) ENGINE = MergeTree()
    ORDER BY col_uint64
">>,
{ok, _} = clickhouse_erl:query(Conn, CreateSQL),

SQL = <<"INSERT INTO type_examples VALUES">>,
Input = [
    #{name => <<"col_uint8">>, type => <<"UInt8">>, data => [0, 255]},
    #{name => <<"col_uint16">>, type => <<"UInt16">>, data => [0, 65535]},
    #{name => <<"col_uint32">>, type => <<"UInt32">>, data => [0, 4294967295]},
    #{name => <<"col_uint64">>, type => <<"UInt64">>, data => [1, 2]},
    #{name => <<"col_int8">>, type => <<"Int8">>, data => [-128, 127]},
    #{name => <<"col_int16">>, type => <<"Int16">>, data => [-32768, 32767]},
    #{name => <<"col_int32">>, type => <<"Int32">>, data => [-2147483648, 2147483647]},
    #{name => <<"col_int64">>, type => <<"Int64">>, data => [-9223372036854775808, 9223372036854775807]},
    #{name => <<"col_float32">>, type => <<"Float32">>, data => [3.14, -2.71]},
    #{name => <<"col_float64">>, type => <<"Float64">>, data => [3.141592653589793, -2.718281828459045]},
    #{name => <<"col_string">>, type => <<"String">>, data => [<<"Hello">>, <<"World">>]},
    #{name => <<"col_bool">>, type => <<"Bool">>, data => [true, false]},
    #{name => <<"col_date">>, type => <<"Date">>, data => [{2023, 1, 1}, {2023, 12, 31}]},
    #{name => <<"col_datetime">>, type => <<"DateTime">>, 
      data => [{{2023, 1, 1}, {0, 0, 0}}, {{2023, 1, 1}, {23, 59, 59}}}},
    #{name => <<"col_datetime64">>, type => <<"DateTime64(6)">>, 
      data => [{{2023, 1, 1}, {0, 0, 0, 0}}, {{2023, 12, 31}, {23, 59, 59, 999999}}]}
],

{ok, #{rows_inserted := 2}} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Connection Pool Management

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Manages a pool of ClickHouse connections to improve performance and resource utilization. Provides functions to start a pool, checkout, and checkin connections.

```erlang
-module(connection_pool_example).
-export([start_pool/2, insert_with_pool/2]).

-record(pool, {
    connections :: [pid()],
    available :: [pid()],
    in_use :: #{pid() => reference()}
}).

start_pool(Size, ConnectOpts) ->
    Connections = [begin
        {ok, Conn} = clickhouse_erl:connect("localhost", 9000, ConnectOpts),
        Conn
    end || _ <- lists:seq(1, Size)],
    
    #pool{
        connections = Connections,
        available = Connections,
        in_use = #{}
    }.

checkout_connection(#pool{available = [Conn | Rest]}) ->
    Ref = make_ref(),
    {ok, Conn, #pool{
        available = Rest,
        in_use = maps:put(Conn, Ref, #pool.in_use)
    }};
checkout_connection(#pool{available = []}) ->
    {error, no_connections_available}.

checkin_connection(Conn, #pool{available = Avail, in_use = InUse}) ->
    #pool{
        available = [Conn | Avail],
        in_use = maps:remove(Conn, InUse)
    }.

insert_with_pool(Pool, InsertFun) ->
    case checkout_connection(Pool) of
        {ok, Conn, NewPool} ->
            try
                Result = InsertFun(Conn),
                {Result, checkin_connection(Conn, NewPool)}
            catch
                _:Error ->
                    {{error, Error}, checkin_connection(Conn, NewPool)}
            end;
        {error, Reason} ->
            {{error, Reason}, Pool}
    end.
```

--------------------------------

### Example: Test Int32 Roundtrip

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/GENERATOR_USAGE.md

Tests the roundtrip encoding and decoding of 32-bit integers using `int32_gen`. Verifies that the integer remains unchanged after serialization and deserialization.

```erlang
prop_int32_roundtrip() ->
    ?FORALL(N, int32_gen(),
        begin
            Encoded = encode_int32(N),
            {ok, Decoded, <<>>} = decode_int32(Encoded),
            N =:= Decoded
        end).
```

--------------------------------

### Specify Correct Types in Placeholders

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_parameters.md

Ensure type safety by specifying the correct data types for placeholders in your queries. Examples show annotations for Float64 and UInt8.

```erlang
% Correct type annotation
#{parameters => [{<<"price">>, <<"99.99">>}]}  % {price:Float64}
```

```erlang
#{parameters => [{<<"active">>, <<"1">>}]}     % {active:UInt8}
```

--------------------------------

### Handle Version Compatibility with Parameters

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/query_parameters.md

This function demonstrates how to handle version compatibility by falling back to a non-parameterized query if parameters are unsupported.

```erlang
execute_with_params(Conn, SQL, Params) ->
    case clickhouse_erl:query(Conn, SQL, #{parameters => Params}) of
        {error, {parameters_unsupported, _}} ->
            % Fallback to non-parameterized query
            execute_without_params(Conn, SQL, Params);
        Result ->
            Result
    end.
```

--------------------------------

### Invalid Column Name Error Example

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/error_handling.md

Illustrates the error for providing a column name that is not a binary string during an INSERT operation. Column names must be binary.

```erlang
Input = [
    #{name => "id", type => @"UInt64", data => [1, 2]}  % String instead of binary
],
{error, {invalid_column_name, "id"}} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Represent Optional Fields in Erlang

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

Erlang code examples show the transformation for nullable fields, using {value, ...} for present values and {null} for absent ones.

```erlang
% Before
{1, <"Alice">, <"alice@example.com">}
```

```erlang
% After
{1, <"Alice">, {value, <"alice@example.com">}}
{2, <"Bob">, {null}}  % Can now represent missing email
```

--------------------------------

### Example: Test Date Roundtrip

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/GENERATOR_USAGE.md

Tests the roundtrip encoding and decoding of Date values using `date_gen`. Verifies that Date objects are preserved correctly through serialization and deserialization.

```erlang
prop_date_roundtrip() ->
    ?FORALL(Date, date_gen(),
        begin
            Encoded = encode_date(Date),
            {ok, Decoded, <<>>} = decode_date(Encoded),
            Date =:= Decoded
        end).
```

--------------------------------

### Basic INSERT with ClickHouse Erlang

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Demonstrates a basic INSERT operation into an 'events' table. It first creates the table if it doesn't exist and then inserts two rows with specified timestamp, event type, user ID, and value.

```erlang
% Create table
CreateSQL = <<"
    CREATE TABLE IF NOT EXISTS events (
        timestamp DateTime,
        event_type String,
        user_id UInt64,
        value Float64
    ) ENGINE = MergeTree()
    ORDER BY timestamp
">>,
{ok, _} = clickhouse_erl:query(Conn, CreateSQL),

% Insert data
SQL = <<"INSERT INTO events (timestamp, event_type, user_id, value) VALUES">>,
Input = [
    #{name => <<"timestamp">>, type => <<"DateTime">>, 
      data => [{{2023, 1, 1}, {12, 0, 0}}, {{2023, 1, 1}, {12, 5, 0}}}},
    #{name => <<"event_type">>, type => <<"String">>, 
      data => [<<"click">>, <<"view">>]},
    #{name => <<"user_id">>, type => <<"UInt64">>, 
      data => [1001, 1002]},
    #{name => <<"value">>, type => <<"Float64">>, 
      data => [1.5, 2.3]}
],

{ok, #{rows_inserted := 2}} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Type Mismatch Error Example for INSERT

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/error_handling.md

Shows the error when data provided for an INSERT column does not match the specified ClickHouse data type. This ensures data integrity.

```erlang
Input = [
    #{name => @"id", type => @"UInt64", data => [1, @"not_a_number", 3]}
],
{error, {type_mismatch, @"id", @"UInt64", Reason}} = 
    clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Row Count Mismatch Error Example

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/error_handling.md

Demonstrates the error returned when columns in an INSERT statement have differing numbers of rows. This check is performed before data is sent to the server.

```erlang
Input = [
    #{name => @"id", type => @"UInt64", data => [1, 2, 3]},
    #{name => @"name", type => @"String", data => [@"Alice", @"Bob"]}  % Only 2 values
],
{error, {row_count_mismatch, [{@"id", 3}, {@"name", 2}]}} = 
    clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### ETS Batch Reader Function

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/streaming-insert.md

A helper function to read a specified number of rows from an ETS table, starting from a given key. It returns the data and the next key for pagination.

```erlang
%% Read up to BatchSize rows from ETS starting at Key. 
read_batch(Key, BatchSize) ->
    read_batch(Key, BatchSize, [], [], []).

read_batch('$end_of_table', _, Ids, Names, Scores) ->
    {lists:reverse(Ids), lists:reverse(Names), lists:reverse(Scores), '$end_of_table'};
read_batch(_Key, 0, Ids, Names, Scores) ->
    {lists:reverse(Ids), lists:reverse(Names), lists:reverse(Scores), _Key};
read_batch(Key, N, Ids, Names, Scores) ->
    [{Id, Name, Score}] = ets:lookup(my_data, Key),
    Next = ets:next(my_data, Key),
    read_batch(Next, N - 1, [Id | Ids], [Name | Names], [Score | Scores]).
```

--------------------------------

### Basic INSERT Syntax

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/insert_quick_reference.md

Demonstrates the basic syntax for inserting data into a ClickHouse table using clickhouse_erl. Requires a connection, the SQL statement, and a list of column definitions with their data.

```erlang
SQL = <<"INSERT INTO table_name (col1, col2, col3) VALUES">>,
Input = [
    #{name => <<"col1">>, type => <<"Type1">>, data => [Val1, Val2, ...]},
    #{name => <<"col2">>, type => <<"Type2">>, data => [Val1, Val2, ...]},
    #{name => <<"col3">>, type => <<"Type3">>, data => [Val1, Val2, ...]}
],
{ok, Result} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### DDL Query Example

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/usage.md

DDL queries and some other query types do not produce DATA packets, thus callbacks are never invoked. Ensure your query returns data if you expect a callback.

```erlang
% Queries that don't invoke callbacks:
PreparedRequest = #{
    sql => <<"CREATE TABLE test (id UInt32)">>,  % DDL - no data
    on_data => Callback  % Never invoked
}
```

--------------------------------

### Run Property Tests with Rebar3

Source: https://github.com/couchemar/clickhouse_erl/blob/main/README.md

Execute property-based tests using the rebar3 build tool and the 'proper' library. This helps ensure code correctness under various conditions.

```bash
# Property tests
rebar3 proper
```

--------------------------------

### Migrate to Pure PropEr Tests

Source: https://github.com/couchemar/clickhouse_erl/blob/main/test/README.md

Use this bash script to migrate existing EUnit-wrapped property tests to the pure PropEr format. This involves copying the file, removing EUnit includes and wrappers, and then testing with `rebar3 proper`.

```bash
# 1. Create new pure property test file
cp clickhouse_erl_compression_property_tests.erl prop_clickhouse_erl_compression.erl

# 2. Remove EUnit include and wrappers
# 3. Keep all prop_* functions unchanged
# 4. Test with rebar3 proper
rebar3 proper --module=prop_clickhouse_erl_compression

# 5. Verify all properties pass
# 6. Delete old file
rm clickhouse_erl_compression_property_tests.erl
```

--------------------------------

### INSERT with Partial Columns using ClickHouse Erlang

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Demonstrates inserting data into a 'users' table, specifying only a subset of columns ('id' and 'name'). Other columns like 'email' and 'created' will use their default values.

```erlang
% Table has columns: id, name, email, created (with DEFAULT now())
% We only insert id and name - email and created use defaults

SQL = <<"INSERT INTO users (id, name) VALUES">>,
Input = [
    #{name => <<"id">>, type => <<"UInt64">>, data => [1, 2, 3]},
    #{name => <<"name">>, type => <<"String">>, 
      data => [<<"Alice">>, <<"Bob">>, <<"Charlie">>]]

{ok, #{rows_inserted := 3}} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Empty Insert (0 rows)

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/insert_quick_reference.md

Demonstrates how to perform an insert operation when no rows are to be inserted. The 'data' lists for all columns should be empty. The result should indicate 0 rows inserted.

```erlang
Input = [
    #{name => <<"id">>, type => <<"UInt64">>, data => []},
    #{name => <<"name">>, type => <<"String">>, data => []}
],
{ok, #{rows_inserted := 0}} = clickhouse_erl:insert(Conn, SQL, Input).
```

--------------------------------

### Connect with LZ4 Compression and Level

Source: https://github.com/couchemar/clickhouse_erl/blob/main/README.md

Establishes a connection to ClickHouse using LZ4 compression with a specified compression level.

```erlang
% LZ4HC with level
{ok, Conn} = clickhouse_erl:connect("localhost", 9000, \
    compression => lz4,
    compression_level => 9
}).
```

--------------------------------

### Execute SELECT with Parameters and Formatting

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/examples.md

Execute a SELECT query with a WHERE clause and ORDER BY, then process the results. This snippet demonstrates accessing columns, rows, and statistics from the query result.

```erlang
SQL = <<"SELECT name, age FROM users WHERE age > 21 ORDER BY age DESC LIMIT 10">>,
{ok, Result} = clickhouse_erl:query(Conn, SQL),

#{
    columns := Columns,
    rows := Rows,
    statistics := Stats
} = Result,

io:format("Found ~p users~n", [length(Rows)]).
```

--------------------------------

### Execute a Simple Query

Source: https://github.com/couchemar/clickhouse_erl/blob/main/README.md

Execute a basic SQL query and retrieve the result.

```erlang
{ok, Result} = clickhouse_erl:query(Conn, <<"SELECT version()">>).
```

--------------------------------

### Basic Streaming Query with on_data Callback

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/usage.md

Use an `on_data` callback to process each data block as it arrives, essential for large result sets. This example shows incremental row counting.

```erlang
PreparedRequest = #{
    sql => <<"SELECT * FROM large_table">>,
    query_id => <<"streaming-query">>,
    on_data => fun(DataBlock, Acc) ->
        % Process data block incrementally
        RowCount = maps:get(rows, DataBlock, 0),
        NewCount = Acc + RowCount,
        io:format("Processed ~p rows (total: ~p)~n", [RowCount, NewCount]),
        {ok, NewCount}
    end,
    initial_accumulator => 0
},
{ok, Result} = clickhouse_erl_connection:query(Conn, PreparedRequest),
TotalRows = maps:get(data, Result).
```

--------------------------------

### Monitoring Query Progress and Profiling

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/usage.md

Include `on_progress` and `on_profile` callbacks in `PreparedRequest` to monitor query execution status and performance metrics.

```erlang
PreparedRequest = #{
    sql => <<"SELECT * FROM large_table">>,
    query_id => <<"monitored-query">>,
    on_data => DataCallback,
    on_progress => fun(ProgressInfo) ->
        RowsRead = maps:get(rows_read, ProgressInfo, 0),
        io:format("Progress: ~p rows read~n", [RowsRead]),
        ok
    end,
    on_profile => fun(ProfileInfo) ->
        io:format("Profile: ~p~n", [ProfileInfo]),
        ok
    end
}.
```

--------------------------------

### Inserting Data into Array Columns

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/composite_types_guide.md

Provides an Erlang example of inserting data into a table with Array columns using the clickhouse_erl:insert function. Ensure the Conn variable is properly initialized.

```erlang
{ok, _} = clickhouse_erl:insert(Conn, <|"posts">|>, [
    {1, <|"First Post">|>, [<|"erlang">|, <|"clickhouse">|>], [100, 150, 200]},
    {2, <|"Second Post">|>, [<|"database">|, <|"performance">|>], [50, 75]},
    {3, <|"Third Post">|>, [<|"tutorial">|>], [300]}
]).
```

--------------------------------

### Connecting to ClickHouse

Source: https://github.com/couchemar/clickhouse_erl/blob/main/doc/usage.md

Establishes a connection to a ClickHouse server using default or custom options.

```APIDOC
## Connecting to ClickHouse

Use `clickhouse_erl:connect/2` or `clickhouse_erl:connect/3` to establish a connection.

### Default Connection
```erlang
{ok, Conn} = clickhouse_erl:connect("localhost", 9000).
```

### Connection with Options
```erlang
Options = #{
    database => <<"my_database">>,
    username => <<"my_user">>,
    password => <<"my_password">>,
    timeout => 5000 % Connection timeout in ms
},
{ok, Conn} = clickhouse_erl:connect("localhost", 9000, Options).
```
```