### Install and Load Parquet Extension in DuckDBex Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Instructions for installing and loading the Parquet extension in DuckDBex, enabling the use of Parquet files. ```elixir # download parquet extension (the .so/.dll extension will be downloaded from the remote source) Duckdbex.query conn, "INSTALL 'parquet';" # load parquet extension into the app Duckdbex.query conn, "LOAD 'parquet';" ``` -------------------------------- ### Local Installation of Unsigned DuckDB Extension Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Demonstrates the process of configuring DuckDB to allow unsigned extensions, creating a connection with this configuration, and then installing and loading a local unsigned extension file. ```elixir # configure DuckDB to allow unsigned extensions {:ok, config} = Duckdbex.create_config() :ok = Duckdbex.set_config_option(config, "allow_unsigned_extensions", true) {:ok, db} = Duckdbex.open(config) {:ok, conn} = Duckdbex.connection(db) # install unsigned extension from local source {:ok, _} = Duckdbex.query(conn, "INSTALL '/home/extensions/my_custom.duckdb_extension';") {:ok, _} = Duckdbex.query(conn, "LOAD '/home/extensions/my_custom.duckdb_extension';") ``` -------------------------------- ### Get Configuration Options Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Retrieves a list of all available configuration options reported by DuckDB. ```APIDOC ## Get Configuration Options ### Description Retrieves a list of all available configuration options reported by DuckDB. ### Function `Duckdbex.get_config_options()` ### Return Value - `list`: A list of maps, where each map contains details about a configuration option (name, type, description, default, etc.). ### Examples ```elixir Duckdbex.get_config_options() # => [ # %{name: "threads", type: :bigint, parameter_type: "BIGINT", description: "...", scope: :global_default, default: 1}, # ... # ] ``` ``` -------------------------------- ### Installing and Loading Remote DuckDB Extension Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Provides the Elixir code to install a verified, signed extension (e.g., 'parquet') from a remote source and then load it into the current connection for use. ```elixir # download extension binary {:ok, _} = Duckdbex.query(conn, "INSTALL 'parquet';") # load extension into the app {:ok, _} = Duckdbex.query(conn, "LOAD 'parquet';") # now you can use the extension {_, r} = Duckdbex.query(conn, "SELECT * FROM 'test.parquet';") ``` -------------------------------- ### Get Available DuckDB Configuration Options Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Retrieves a list of all available configuration options reported by DuckDB, including their name, type, scope, and default values. ```elixir Duckdbex.get_config_options() # => [ # %{ ... } ``` -------------------------------- ### Read Parquet Files with DuckDBex Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Examples of reading single or multiple Parquet files using glob patterns, and describing the schema of a Parquet file. ```elixir # read a single parquet file Duckdbex.query conn, "SELECT * FROM 'test.parquet';" # figure out which columns/types are in a parquet file Duckdbex.query conn, "DESCRIBE SELECT * FROM 'test.parquet';" # read all files that match the glob pattern Duckdbex.query conn, "SELECT * FROM 'test/*.parquet';" ``` -------------------------------- ### Load and Use Custom DuckDB Extension in Elixir Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Configure Duckdbex to allow unsigned extensions, open a database connection, and then install and load the custom extension before using its functions. ```elixir {:ok, config} = Duckdbex.create_config() :ok = Duckdbex.set_config_option(config, "allow_unsigned_extensions", true) {:ok, db} = Duckdbex.open(config) {:ok, conn} = Duckdbex.connection(db) # load extension into the app {:ok, _} = Duckdbex.query(conn, "INSTALL '/home/duckdb_unsigned_extensions/json.duckdb_extension';") {:ok, _} = Duckdbex.query(conn, "LOAD '/home/duckdb_unsigned_extensions/json.duckdb_extension';") # use the extension {:ok, _} = Duckdbex.query(conn, "SELECT * FROM read_json_objects('some_data.json');") ... ``` -------------------------------- ### Add Duckdbex Dependency to Mix.exs Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Add the duckdbex package to your project's dependencies in mix.exs to install it. ```elixir def deps do [ {:duckdbex, "~> 0.4.0"} ] end ``` -------------------------------- ### Configure DuckDB Options Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Creates a configuration reference, sets a specific option like 'checkpoint_threshold', and then opens the database with this configuration. ```elixir {:ok, config} = Duckdbex.create_config() :ok = Duckdbex.set_config_option(config, "checkpoint_threshold", "8MB") {:ok, _db} = Duckdbex.open(config) ``` -------------------------------- ### Create Configuration Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Creates a configuration reference for DuckDB. Options can be set on this config and then passed to `Duckdbex.open/1` or `Duckdbex.open/2`. ```APIDOC ## Create Configuration ### Description Creates a configuration reference for DuckDB. Options can be set on this config and then passed to `Duckdbex.open/1` or `Duckdbex.open/2`. ### Function `Duckdbex.create_config()` ### Return Value - `{:ok, config_reference}`: On success, returns an OK tuple with a configuration reference. - `{:error, reason}`: On failure, returns an error tuple. ### Examples ```elixir {:ok, config} = Duckdbex.create_config() ``` ``` -------------------------------- ### Export Database to Parquet Format with DuckDBex Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Demonstrates how to export the entire database contents to a specified directory in Parquet format using DuckDBex. ```elixir # export the table contents of the entire database as parquet Duckdbex.query conn, "EXPORT DATABASE 'target_directory' (FORMAT PARQUET);" ``` -------------------------------- ### Fetch All Results from Query Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Run a query and retrieve all results at once using the result reference. Suitable for smaller result sets. ```elixir # Run a query and get the reference to the result {:ok, result_ref} = Duckdbex.query(conn, """ SELECT userId, movieId, rating FROM ratings WHERE userId = $1; """, [1]) # Get all the data from the result reference at once Duckdbex.fetch_all(result_ref) # => [[userId: 1, movieId: 1, rating: 6], [userId: 1, movieId: 2, rating: 12]] ``` -------------------------------- ### Prepare and Execute a Statement Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Prepare a SQL statement for repeated execution with different parameters. This improves performance and prevents SQL injection. Results from execution can then be fetched. ```elixir # Prepare statement {:ok, stmt_ref} = Duckdbex.prepare_statement(conn, "SELECT 1;") # Execute statement {:ok, result_ref} = Duckdbex.execute_statement(stmt_ref) # Fetch result Duckdbex.fetch_all(result_ref) # => [[1]] ``` ```elixir # Prepare statement {:ok, stmt_ref} = Duckdbex.prepare_statement(conn, "SELECT * FROM ratings WHERE userId = $1;") # Execute statement {:ok, result_ref} = Duckdbex.execute_statement(stmt_ref, [1]) # fetch result ... # Execute statement {:ok, result_ref} = Duckdbex.execute_statement(stmt_ref, [42]) # fetch result ... ``` -------------------------------- ### Work with LIST Data Type in DuckDBex Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Shows how to select a list directly and how to create and insert data into a table with columns of LIST type in DuckDBex. ```elixir # Just select a list {:ok, r} = Duckdbex.query(conn, "SELECT [1, 2, 3];") [[[1, 2, 3]]] = Duckdbex.fetch_all(r) # Table with columns of LIST type Duckdbex.query(conn, "CREATE TABLE list_table (int_list INT[], varchar_list VARCHAR[]);") Duckdbex.query(conn, "INSERT INTO list_table VALUES ([1, 2], ['one', 'two']), ([4, 5], ['three', NULL]), ([6, 7], NULL);") {:ok, r} = Duckdbex.query(conn, "SELECT * FROM list_table") [[[1, 2], ["one", "two"]], [[4, 5], ["three", nil]], [[6, 7], nil]] = Duckdbex.fetch_all(r) ``` -------------------------------- ### Export Data to CSV with DuckDBex Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Shows how to create a table from a CSV file and then export query results to a new CSV file using DuckDBex. Supports specifying headers and delimiters. ```elixir Duckdbex.query conn, "CREATE TABLE ontime AS SELECT * FROM 'test.csv';" # write the result of a query to a CSV file Duckdbex.query conn, "COPY (SELECT * FROM ontime) TO 'test.csv' WITH (HEADER 1, DELIMITER '|');" ``` -------------------------------- ### Build Custom DuckDB Extension Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Clone the DuckDB repository, build the extension using make, and copy the compiled extension to a designated directory for unsigned extensions. ```shell $ git clone https://github.com/duckdb/duckdb.git duckdb$ cd duckdb duckdb$ BUILD_JSON=1 make duckdb$ cp build/release/extension/json/json.duckdb_extension /home/duckdb_unsigned_extensions ``` -------------------------------- ### Load CSV File into DuckDBex Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Demonstrates how to read a CSV file directly from disk into a DuckDB database using DuckDBex. Auto-inference of CSV options is supported. ```elixir {:ok, db} = Duckdbex.open() {:ok, conn} = Duckdbex.connection(db) # read a CSV file from disk, auto-infer options {:ok, res} = Duckdbex.query conn, "SELECT * FROM 'test.csv';" Duckdbex.fetch_all(res) # will result in [ [{1988, 1, 1}, "AA", "New York, NY", "Los Angeles, CA"], [{1988, 1, 2}, "AA", "New York, NY", "Los Angeles, CA"], [{1988, 1, 3}, "AA", "New York, NY", "Los Angeles, CA"] ] ``` -------------------------------- ### Open an Existing DuckDB Database File Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Opens an existing DuckDB database file. If the file does not exist, it will be created. ```elixir # Open an existing database file {:ok, db} = Duckdbex.open("exis_movies.duckdb") ``` -------------------------------- ### Configuring Extension Directory in DuckDBEx Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Shows how to set the extension directory configuration option for DuckDB within Duckdbex, either through connection configuration or by executing a SET command. ```elixir SET extension_directory = '/path/to/your/extension/directory'; ``` -------------------------------- ### Open Database Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Opens a DuckDB database file. If the file does not exist, it will be created. An in-memory database can be created by omitting the file path. ```APIDOC ## Open Database ### Description Opens a DuckDB database file. If the file does not exist, it will be created. An in-memory database can be created by omitting the file path. ### Function `Duckdbex.open(file_path \\ :memory)` ### Parameters - `file_path` (string, optional): The path to the DuckDB database file. Defaults to `:memory` for an in-memory database. ### Return Value - `{:ok, db_reference}`: On success, returns an OK tuple with a database reference. - `{:error, reason}`: On failure, returns an error tuple. ### Examples ```elixir # Open an existing database file {:ok, db} = Duckdbex.open("exis_movies.duckdb") # Create a new database file if it does not exist {:ok, db} = Duckdbex.open("not_yet_exist_my_new_duckdb_database") # Create a database in memory {:ok, db} = Duckdbex.open() ``` ``` -------------------------------- ### Create a DuckDB Connection Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Creates a connection to an opened DuckDB database instance. Each connection is a native OS thread and should ideally be used from different Elixir processes for optimal performance. ```elixir {:ok, db} = Duckdbex.open() # Create a connection to the opened database {:ok, conn} = Duckdbex.connection(db) ``` -------------------------------- ### Create a New DuckDB Database File Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Creates a new DuckDB database file if the specified file does not exist. ```elixir # If the specified file does not exist the new database will be created {:ok, db} = Duckdbex.open("not_yet_exist_my_new_duckdb_database") ``` -------------------------------- ### Create an In-Memory DuckDB Database Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Creates a DuckDB database in memory. Data is not persisted and will be lost when the database connection is closed or goes out of scope. ```elixir # Just create database in the memory {:ok, db} = Duckdbex.open() ``` -------------------------------- ### Using MAP Data Type in DuckDBEx Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Shows how to create a table with a MAP column, insert data into it, and query the data. MAPs allow for flexible key-value storage where keys do not need to be consistent across rows. ```elixir Duckdbex.query(conn, "CREATE TABLE map_table (map_col MAP(INT, DOUBLE));") Duckdbex.query(conn, "INSERT INTO map_table VALUES (map([1, 2], [2.98, 3.14])), (map([3, 4], [9.8, 1.6]));") {:ok, r} = Duckdbex.query(conn, "SELECT * FROM map_table") [[[{1, 2.98}, {2, 3.14}]], [[{3, 9.8}, {4, 1.6}]]] = Duckdbex.fetch_all(r) ``` -------------------------------- ### Execute a SQL Query with Parameters Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Executes a SQL query with parameters, passing the parameters as a list. This is useful for preventing SQL injection and for dynamic query construction. ```elixir {:ok, db} = Duckdbex.open() {:ok, conn} = Duckdbex.connection(db) # Run a query with parameters {:ok, result_ref} = Duckdbex.query(conn, "SELECT 1 WHERE $1 = 1;", [1]) ``` -------------------------------- ### Open a DuckDB Database Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Opens an existing DuckDB database file. The returned reference must be managed for automatic resource cleanup or explicitly released. ```elixir # Open an existing database file {:ok, db} = Duckdbex.open("exis_movies.duckdb") #Reference<0.1076596279.3008626690.232411> ``` -------------------------------- ### Insert Data using SQL Statements Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Use standard SQL INSERT statements to add data to tables. This method is recommended for small numbers of records; for bulk inserts, consider using the Appender. ```elixir {:ok, db} = Duckdbex.open() {:ok, conn} = Duckdbex.connection(db) {:ok, _res} = Duckdbex.query(conn, "CREATE TABLE people(id INTEGER, name VARCHAR);") {:ok, _res} = Duckdbex.query(conn, "INSERT INTO people VALUES (1, 'Mark'), (2, 'Hannes');") ``` -------------------------------- ### Append Data to DuckDB Table with DuckDBex Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Illustrates how to use the Appender to efficiently load bulk data into a DuckDB table. Rows are cached for performance and can be flushed manually or automatically. ```elixir {:ok, db} = Duckdbex.open() {:ok, conn} = Duckdbex.connection(db) Duckdbex.query(conn, "CREATE TABLE people(id INTEGER, name VARCHAR);") # Create Appender for table 'people' {:ok, appender} = Duckdbex.appender conn, "people" # Append two rows at once. Duckdbex.appender_add_rows appender, [[2, "Sarah"], [3, "Alex"]] # Persist cached rows Duckdbex.appender_flush() ``` -------------------------------- ### Create Connection Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Creates a connection to an opened DuckDB database instance. Multiple connections can be created for parallel processing. ```APIDOC ## Create Connection ### Description Creates a connection to an opened DuckDB database instance. Multiple connections can be created for parallel processing. ### Function `Duckdbex.connection(db_reference)` ### Parameters - `db_reference`: The reference to the opened DuckDB database instance. ### Return Value - `{:ok, connection_reference}`: On success, returns an OK tuple with a connection reference. - `{:error, reason}`: On failure, returns an error tuple. ### Examples ```elixir {:ok, db} = Duckdbex.open() {:ok, conn} = Duckdbex.connection(db) ``` ``` -------------------------------- ### Execute a SQL Query Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Executes a SQL query against a DuckDB connection and returns a reference to the result set. ```elixir {:ok, db} = Duckdbex.open() {:ok, conn} = Duckdbex.connection(db) # Run a query and get the reference to the result {:ok, result_ref} = Duckdbex.query(conn, "SELECT 1;") ``` -------------------------------- ### Set Configuration Option Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Sets a specific configuration option for a given DuckDB configuration. ```APIDOC ## Set Configuration Option ### Description Sets a specific configuration option for a given DuckDB configuration. ### Function `Duckdbex.set_config_option(config_reference, option_name, option_value)` ### Parameters - `config_reference`: The reference to the DuckDB configuration. - `option_name` (string): The name of the configuration option to set. - `option_value` (any): The value to set for the configuration option. ### Return Value - `:ok`: On success. - `{:error, reason}`: On failure. ### Examples ```elixir {:ok, config} = Duckdbex.create_config() :ok = Duckdbex.set_config_option(config, "checkpoint_threshold", "8MB") ``` ``` -------------------------------- ### Execute Query Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Executes an SQL query against a DuckDB connection. Supports parameterized queries. ```APIDOC ## Execute Query ### Description Executes an SQL query against a DuckDB connection. Supports parameterized queries. ### Function `Duckdbex.query(connection_reference, query, parameters \\ [])` ### Parameters - `connection_reference`: The reference to the DuckDB connection. - `query` (string): The SQL query string to execute. - `parameters` (list, optional): A list of parameters to be used in the query. Defaults to an empty list. ### Return Value - `{:ok, result_reference}`: On success, returns an OK tuple with a reference to the query result. - `{:error, reason}`: On failure, returns an error tuple. ### Examples ```elixir {:ok, db} = Duckdbex.open() {:ok, conn} = Duckdbex.connection(db) # Run a simple query {:ok, result_ref} = Duckdbex.query(conn, "SELECT 1;") # Run a query with parameters {:ok, result_ref} = Duckdbex.query(conn, "SELECT 1 WHERE $1 = 1;", [1]) ``` ``` -------------------------------- ### Working with UNION Data Type in DuckDBEx Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Illustrates creating a table with a UNION column, inserting values of different types, and querying them. The UNION type stores a single value from a set of possible types, identified by a tag. ```elixir {:ok, _} = Duckdbex.query(conn, "CREATE TABLE tbl1(u UNION(num INT, str VARCHAR));") {:ok, _} = Duckdbex.query(conn, "INSERT INTO tbl1 values (1) , ('two') , (union_value(str := 'three'));") {:ok, r} = Duckdbex.query(conn, "SELECT u from tbl1;") [[{"num", 1}], [{"str", "two"}], [{"str", "three"}]] = Duckdbex.fetch_all(r) ``` -------------------------------- ### ENUM Data Type Operations in DuckDBEx Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Demonstrates creating an ENUM type, a table using that type, inserting data, and querying based on ENUM values. ENUMs are efficient for columns with a fixed, known set of string values. ```elixir {:ok, _} = Duckdbex.query(conn, "CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');") {:ok, _} = Duckdbex.query(conn, "CREATE TABLE person (name text, current_mood mood);") {:ok, _} = Duckdbex.query(conn, "INSERT INTO person VALUES ('Pedro','happy'), ('Mark', NULL), ('Pagliacci', 'sad'), ('ackey', 'ok');") {:ok, r} = Duckdbex.query(conn, "SELECT * FROM person WHERE current_mood = 'sad';") [["Pagliacci", "sad"]] = Duckdbex.fetch_all(r) {:ok, r} = Duckdbex.query(conn, "SELECT * FROM person WHERE current_mood = $1;", ["sad"]) [["Pagliacci", "sad"]] = Duckdbex.fetch_all(r) ``` -------------------------------- ### Fetch Results in Chunks Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Retrieve data from a query result incrementally using `fetch_chunk/1`. This is memory-efficient for large datasets. Repeated calls fetch subsequent chunks until the result set is exhausted, at which point `fetch_chunk/1` returns an empty list. ```elixir # Run a query and get the reference to the result {:ok, result_ref} = Duckdbex.query(conn, "SELECT * FROM ratings;") # fetch chunk Duckdbex.fetch_chunk(result_ref) # => [[userId: 1, movieId: 1, rating: 6], [userId: 1, movieId: 2, rating: 12]...] # fetch next chunk Duckdbex.fetch_chunk(result_ref) # => [] ... # the data is over and fetch_chunk returns the empty list Duckdbex.fetch_chunk(result_ref) # => [] ``` -------------------------------- ### Querying STRUCT Data Type in DuckDBEx Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Demonstrates querying a STRUCT literal and fetching the result. The result is a list containing a map representing the STRUCT. ```elixir {:ok, r} = Duckdbex.query(conn, "SELECT {'x': 1, 'y': 2, 'z': 3};") [[%{"x" => 1, "y" => 2, "z" => 3}]] = Duckdbex.fetch_all(r) ``` -------------------------------- ### Explicitly Release DuckDB Resources Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md Manually close and release database, connection, or result references. This is useful for ensuring resources are freed promptly, especially when automatic garbage collection is not immediately desired. It flushes buffers to disk before closing. ```elixir iex> {:ok, db} = Duckdbex.open("my_database.duckdb") iex> {:ok, conn} = Duckdbex.connection(db) iex> {:ok, res} = Duckdbex.query(conn, "SELECT 1 WHERE $1 = 1;", [1]) iex> :ok = Duckdbex.release(res) iex> :ok = Duckdbex.release(conn) iex> :ok = Duckdbex.release(db) ``` -------------------------------- ### Convert HUGEINT to Elixir Integer Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md When retrieving HUGEINT values from DuckDB, they are returned as a tuple of two integers. Use `Duckdbex.hugeint_to_integer/1` to convert this tuple into a single Elixir integer. ```elixir > {:ok, r} = Duckdbex.query(conn, "SELECT SUM(1);") > Duckdbex.fetch_all(r) [[{0, 1}]] > Duckdbex.hugeint_to_integer({0, 1}) 1 ``` -------------------------------- ### Convert Elixir Integer to HUGEINT for SQL Query Source: https://github.com/alexr2d2/duckdbex/blob/main/README.md To pass a large integer as a HUGEINT argument in a SQL query, first convert the Elixir integer to DuckDB's native HUGEINT representation using `Duckdbex.integer_to_hugeint/1`. ```elixir > hi = Duckdbex.integer_to_hugeint(123456789123456789123456789) {6692605, 17502027875430457109} > {:ok, r} = Duckdbex.query(conn, "SELECT SUM(1234567891234567891234567891) > $1;", [hi]) > Duckdbex.fetch_all(r) [[true]] ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.