### Install Python dependencies with pip Source: https://github.com/delta-io/delta-rs/blob/main/docs/index.md Installs the required Python packages for the quick start example. pyarrow and pandas are needed for DataFrame import, and tabulate is needed to print the DataFrame in the final example. ```bash pip install deltalake pandas pyarrow tabulate ``` -------------------------------- ### Setup Data for Delete Operation Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/merging-tables.md Initializes target and source PyArrow tables for the delete operation example. ```python target_data = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]}) write_deltalake("tmp_table", target_data) dt = DeltaTable("tmp_table") source_data = pa.table({"x": [2, 3], "deleted": [False, True]}) ``` -------------------------------- ### Import deltalake and pandas Source: https://github.com/delta-io/delta-rs/blob/main/docs/index.md Imports the required dependencies for the quick start example. ```python from deltalake import write_deltalake, DeltaTable import pandas as pd ``` -------------------------------- ### Install Python documentation requirements Source: https://github.com/delta-io/delta-rs/blob/main/CONTRIBUTING.md Activates the uv environment and installs the necessary dependencies for building documentation. ```sh cd .. source python/.venv/bin/activate pip install -r docs/requirements.txt ``` -------------------------------- ### Setup Virtual Environment with make setup Source: https://github.com/delta-io/delta-rs/blob/main/python/CONTRIBUTING.md Initializes a local development environment using the Makefile, creating a virtual environment with dependencies managed by uv. Run this first to prepare for development. ```bash make setup ``` -------------------------------- ### Run a Rust example Source: https://github.com/delta-io/delta-rs/blob/main/CONTRIBUTING.md Runs a specific Rust example with the datafusion feature enabled. ```sh cd ../crates/deltalake cargo run --example basic_operations --features="datafusion" ``` -------------------------------- ### Install deltalake with pip Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/installation.md Install the deltalake Python package using pip. ```bash pip install deltalake ``` -------------------------------- ### Set up samply and record a flamegraph Source: https://github.com/delta-io/delta-rs/blob/main/crates/benchmarks/README.md Install samply, build the profiled binary, then record a flamegraph. This is useful for profiling a merge operation. ```bash cargo install samply --locked cargo build --profile profiling -p delta-benchmarks samply record ./target/profiling/delta-benchmarks merge upsert ``` -------------------------------- ### Setup target and source data for merge Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/merging-tables.md Initializes the target Delta table and source data for demonstration purposes. ```python target_data = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]}) write_deltalake("tmp_table", target_data) dt = DeltaTable("tmp_table") source_data = pa.table({"x": [2, 3, 5], "y": [5, 8, 11]}) ``` -------------------------------- ### Build the project for development Source: https://github.com/delta-io/delta-rs/blob/main/CONTRIBUTING.md Builds the project and installs deltalake into the Python virtual environment managed by uv. ```sh cd python make develop ``` -------------------------------- ### Install delta-rs with make develop Source: https://github.com/delta-io/delta-rs/blob/main/python/CONTRIBUTING.md Builds the Python package using maturin and installs it in the current virtual environment. Run after activating the environment. ```bash make develop ``` -------------------------------- ### Install dask-deltatable Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-dask.md Install the dask-deltatable library to enable reading Delta Lake tables into Dask DataFrames. ```bash pip install dask-deltatable ``` -------------------------------- ### Partition-specific optimize output Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/optimize/small-file-compaction-with-optimize.md Example output showing the metrics returned after running optimize on a specific partition. ```python {'numFilesAdded': 1, 'numFilesRemoved': 21, 'filesAdded': {'min': 238282, 'max': 238282, 'avg': 238282.0, 'totalFiles': 1, 'totalSize': 238282}, 'filesRemoved': {'min': 10244, 'max': 39000, 'avg': 11613.333333333334, 'totalFiles': 21, 'totalSize': 243880}, 'partitionsOptimized': 1, 'numBatches': 1, 'totalConsideredFiles': 21, 'totalFilesSkipped': 0, 'plannerStrategy': 'preserveLocality', 'preservedStableOrder': True, 'preserveInsertionOrder': True, 'maxBinSpanFiles': 21} ``` -------------------------------- ### Install dagster-deltalake-polars Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-dagster.md Install the necessary package to use the Delta Lake Polars I/O manager. ```bash pip install dagster-deltalake-polars ``` -------------------------------- ### Activate Virtual Environment with source ./.venv/bin/activate Source: https://github.com/delta-io/delta-rs/blob/main/python/CONTRIBUTING.md Activates the virtual environment created by 'make setup'. Use this before running development commands. ```bash source ./.venv/bin/activate ``` -------------------------------- ### Install Rust compiler for building custom wheels Source: https://github.com/delta-io/delta-rs/blob/main/python/README.md Installs the Rust compiler via rustup, which is required to compile the package. ```sh curl https://sh.rustup.rs -sSf | sh -s ``` -------------------------------- ### Install deltalake with pip, uv, or poetry Source: https://github.com/delta-io/delta-rs/blob/main/python/README.md Installation commands for the deltalake package using pip, uv, and poetry. Note that official binary wheels are linked against openssl statically for remote object store communication. ```bash # with pip pip install deltalake # with uv uv add deltalake # with poetry poetry add deltalake ``` -------------------------------- ### Setup for conditional updates Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/merging-tables.md Initializes target and source data for demonstrating conditional updates on rows not matched by source. ```python target_data = pa.table({"1 x": [1, 2, 3], "1y": [4, 5, 6]}) write_deltalake("tmp", target_data) dt = DeltaTable("tmp") source_data = pa.table({"x": [2, 3, 4]}) ``` -------------------------------- ### Python: Development and testing options for local services Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/object-storage/special_configuration.md Example storage_options for local development with MinIO, LocalStack, or Azurite. Includes allow_http set to 'true' for non-HTTPS endpoints. ```python storage_options = { 'AWS_ACCESS_KEY_ID': 'test', 'AWS_SECRET_ACCESS_KEY': 'test', 'AWS_ENDPOINT_URL': 'http://localhost:4566', 'allow_http': 'true', # Required for non-HTTPS endpoints 'connect_timeout': '10s', 'timeout': '60s', } ``` -------------------------------- ### Install krb5 on macOS Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/object-storage/hdfs.md Required Kerberos library installation for macOS via Homebrew. Needed for Kerberos authentication to HDFS. ```bash brew install krb5 ``` -------------------------------- ### Install libgssapi_krb5 on Debian-based systems Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/object-storage/hdfs.md Required Kerberos library installation for Debian/Ubuntu systems. Without this package, Kerberos authentication to HDFS will not work. ```bash apt-get install libgssapi_krb5-2 ``` -------------------------------- ### Vacuum command output Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/optimize/small-file-compaction-with-optimize.md Example list of files returned by the vacuum command after successful execution. ```python ['date=2021-01-02/39-a98680f2-0e0e-4f26-a491-18b183f9eb05-0.parquet', 'date=2021-01-02/41-e96bc8bb-c571-484c-b534-e897424fb7da-0.parquet', … 'date=2021-01-01/0-cfe227c6-edd9-4369-a1b0-db4559a2e693-0.parquet', 'date=2021-01-01/18-ded53418-172b-4e40-bf2e-7c8142e71bd1-0.parquet'] ``` -------------------------------- ### Basic DeltaTable usage example Source: https://github.com/delta-io/delta-rs/blob/main/python/README.md Shows how to open a Delta table and retrieve its version and file URIs. The output includes the version number and a list of parquet file URIs. ```python from deltalake import DeltaTable dt = DeltaTable("../rust/tests/data/delta-0.2.0") dt.version() 3 dt.file_uris() ['s3://bucket/table/part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet', 's3://bucket/table/part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet', 's3://bucket/table/part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet'] ``` -------------------------------- ### Install krb5-libs on RHEL-based systems Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/object-storage/hdfs.md Required Kerberos library installation for RHEL/CentOS/Fedora systems. This package must be present for Kerberos authentication to function. ```bash yum install krb5-libs ``` -------------------------------- ### Partitioning an asset with StaticPartitionsDefinition Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-dagster.md Example of partitioning the Iris dataset on a specific column using the @asset decorator with partitions_def and metadata. ```python import pyarrow as pa from dagster import StaticPartitionsDefinition, asset @asset( partitions_def=StaticPartitionsDefinition( ["Human", "Horse",] ), metadata={"partition_expr": "n_legs"}, ) def dataset_partitioned( context, clean_dataset: pa.Table, ) -> pa.Table: animals = context.asset_partition_key_for_output() table = clean_dataset return table.filter(pc.field("animals") == animals) ``` -------------------------------- ### Merge Operation Logs With Partition Predicates Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/merging-tables.md Example logs showing an improved merge operation where partition columns are included in the predicates, allowing the engine to skip irrelevant files. ```text Merging table with predicates: { 'merge': 's.unique_constraint_hash = t.unique_constraint_hash AND s.month_id = t.month_id AND t.month_id = 202503', 'when_matched_update_all': 's.post_transform_row_hash != t.post_transform_row_hash AND s.month_id = t.month_id AND t.month_id = 202503' } Files Scanned: 1 Files Skipped: 10 Files Added: 1 Execution Time: 2964ms ``` -------------------------------- ### Prepare vacuum fixture and run Divan bench Source: https://github.com/delta-io/delta-rs/blob/main/crates/benchmarks/README.md Prepare the fixture once (skips if already present at default path) and run the Divan bench for flat and parallel scans at default concurrency. ```bash # 1) Prepare fixture once (skips if already present at default path) cargo run --release -p delta-benchmarks -- generate-vacuum-fixture # 2) Divan bench (flat + parallel at default concurrency) cargo bench -p delta-benchmarks --bench vacuum ``` -------------------------------- ### Get Delta Lake schema with DeltaTable.schema() Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/examining-table.md Retrieve the Delta Lake schema using DeltaTable.schema() in Python or DeltaTable::get_schema (via state.schema()) in Rust. The Python example shows the resulting Schema object with a Field for the id column. ```python >>> from deltalake import DeltaTable >>> dt = DeltaTable("../rust/tests/data/simple_table") >>> dt.schema() Schema([Field(id, PrimitiveType("long"), nullable=True)]) ``` ```rust let delta_path = Url::from_directory_path("/tmp/some-table").unwrap(); let table = open_table(delta_path).await?; let state = table.state.as_ref().unwrap(); let schema = state.schema(); println!("schema: {:?}", schema); ``` -------------------------------- ### Get table metadata with DeltaTable.metadata() Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/examining-table.md Retrieve table metadata using DeltaTable.metadata() in Python or deltalake::open_table and state.metadata() in Rust. The Python example shows the resulting Metadata object with fields like id, name, description, partitionColumns, created_time, and configuration. ```python >>> from deltalake import DeltaTable >>> dt = DeltaTable("../rust/tests/data/simple_table") >>> dt.metadata() Metadata(id: 5fba94ed-9794-4965-ba6e-6ee3c0d22af9, name: None, description: None, partitionColumns: [], created_time: 1587968585495, configuration={}) ``` ```rust let delta_path = Url::from_directory_path("/rust/tests/data/simple_table").unwrap(); let table = deltalake::open_table(delta_path).await?; let state = table.state.as_ref().unwrap(); println!("metadata: {:?}", state.metadata()); ``` -------------------------------- ### Load Delta Table from Unity Catalog in Python Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/loading-table.md Demonstrates how to initialize a DeltaTable instance by setting the necessary environment variables for the workspace URL and access token, then providing the catalog, database, and table names. ```python import os from deltalake import DeltaTable # Set your Unity Catalog workspace URL in the # DATABRICKS_WORKSPACE_URL environment variable. os.environ["DATABRICKS_WORKSPACE_URL"] = "https://adb-1234567890.XX.azuredatabricks.net" # Set your Unity Catalog access token in the # DATABRICKS_ACCESS_TOKEN environment variable. os.environ["DATABRICKS_ACCESS_TOKEN"] = "" # Your Unity Catalog name here catalog_name = "unity" # Your UC database name here db_name = "my_database" # Your table name in the UC database here table_name = "my_table" # Full UC URL in required format uc_full_url = f"{catalog_name}.{db_name}.{table_name}" # This should load the valid delta table at specified location, # and you can start using it. dt = DeltaTable(uc_full_url) dt.is_deltatable(dt.table_uri) # True ``` -------------------------------- ### Read Delta table files with Dask using file_uris Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/querying-delta-tables.md Use DeltaTable.file_uris to get file paths and pass them to dask.dataframe.read_parquet. The file_uris accepts the same predicates for pruning. The example shows both pruned and unpruned reads, displaying the Dask DataFrame structure and compute result. ```python >>> import dask.dataframe as dd >>> df = dd.read_parquet(dt.file_uris(file_pruning_predicate="year = 2021 OR month = 2")) >>> df = dd.read_parquet(dt.file_uris()) >>> df Dask DataFrame Structure: value year month day npartitions=6 object category[known] category[known] category[known] ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... Dask Name: read-parquet, 6 tasks >>> df.compute() value year month day 0 1 2020 1 1 0 2 2020 2 3 0 3 2020 2 5 0 4 2021 4 5 0 5 2021 12 4 0 6 2021 12 20 1 7 2021 12 20 ``` -------------------------------- ### Generate custom vacuum fixture and run bench with custom parameters Source: https://github.com/delta-io/delta-rs/blob/main/crates/benchmarks/README.md Generate a custom fixture with specified days and groups, then run the Divan bench with custom fixture, latency, and sample count. ```bash cargo run --release -p delta-benchmarks -- generate-vacuum-fixture \ --out crates/benchmarks/data/vacuum_bench/custom \ --days 90 --groups 2000 --force cargo bench -p delta-benchmarks --bench vacuum -- \ --fixture crates/benchmarks/data/vacuum_bench/custom \ --list-latency-ms 100 \ --sample-count 2 ``` -------------------------------- ### Installing dagster-deltalake-pandas Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-dagster.md Command to install the necessary package for using Delta Lake with pandas in Dagster. ```bash pip install dagster-deltalake-pandas ``` -------------------------------- ### Initialize Target Table (Python) Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/merging-tables.md Initializes a target table with sample data using Python. ```python > target_data = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]}) > write_deltalake("tmp_table", target_data) > dt = DeltaTable("tmp_table") x y 0 1 4 1 2 5 2 3 6 ``` -------------------------------- ### Install deltalake with conda Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/installation.md Install the deltalake Python package using conda from the conda-forge channel. ```bash conda install -c conda-forge deltalake ``` -------------------------------- ### Install dagster-deltalake Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-dagster.md Install the necessary package to use the Delta Lake I/O manager in your Dagster project. ```bash pip install dagster-deltalake ``` -------------------------------- ### List Available Tasks with make help Source: https://github.com/delta-io/delta-rs/blob/main/python/CONTRIBUTING.md Displays all available tasks defined in the Makefile, helping you discover other commands during development. ```bash make help ``` -------------------------------- ### Python: Example usage of common client options with write_deltalake Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/object-storage/special_configuration.md Shows how to pass common client options like timeout and connect_timeout via storage_options when writing a Delta table to S3. Includes cloud-specific credentials. ```python from deltalake import write_deltalake storage_options = { # Cloud-specific credentials 'AWS_ACCESS_KEY_ID': 'your-key', 'AWS_SECRET_ACCESS_KEY': 'your-secret', 'AWS_REGION': 'us-east-1', # Common client options (work with any backend) 'timeout': '120s', 'connect_timeout': '30s', 'pool_max_idle_per_host': '10', } write_deltalake("s3://bucket/table", df, storage_options=storage_options) ``` -------------------------------- ### Explain Query Plan for Partitioned Column Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-daft.md Running explain on a query filtering by a partitioned column shows that Daft optimizes the scan by filtering partitions. ```text (...) == Optimized Logical Plan == * PythonScanOperator: DeltaLakeScanOperator(None) | File schema = first_name#Utf8, last_name#Utf8, country#Utf8, continent#Utf8 | Partitioning keys = [PartitionField(country#Utf8)] | Partition Filter = col(country) == lit("Germany") | Output schema = first_name#Utf8, last_name#Utf8, country#Utf8, continent#Utf8 == Physical Plan == * TabularScan: | Num Scan Tasks = 1 | Estimated Scan Bytes = 1025 | Clustering spec = { Num partitions = 1 } ``` -------------------------------- ### Get Arrow schema with to_arrow() Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/examining-table.md Retrieve the Arrow schema. In Python, use DeltaTable.schema.to_arrow() to get an Arro3 schema; in Rust, use state.snapshot().arrow_schema(). ```python >>> dt.schema().to_arrow() id: int64 ``` ```rust let arrow_schema = state.snapshot().arrow_schema(); println!("arrow_schema: {:?}", schema); ``` -------------------------------- ### Load DeltaTable with storage_options in Python Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/loading-table.md Shows how to instantiate a DeltaTable with AWS credentials passed via storage_options in Python. ```python storage_options = {"AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY":"THE_AWS_SECRET_ACCESS_KEY"} dt = DeltaTable("../rust/tests/data/delta-0.2.0", storage_options=storage_options) ``` -------------------------------- ### Install Daft with deltalake extras Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-daft.md Install Daft with the deltalake extras using pip to enable Delta Lake support. This adds the deltalake Python package for fetching table metadata. ```python !pip install -U "getdaft[deltalake]" ``` -------------------------------- ### Run Divan bench with environment variable fallbacks Source: https://github.com/delta-io/delta-rs/blob/main/crates/benchmarks/README.md Use environment variables VACUUM_BENCH_FIXTURE and VACUUM_BENCH_LIST_LATENCY_MS as fallbacks instead of passing flags. ```bash VACUUM_BENCH_FIXTURE=crates/benchmarks/data/vacuum_bench/custom \ VACUUM_BENCH_LIST_LATENCY_MS=100 \ cargo bench -p delta-benchmarks --bench vacuum -- --sample-count 1 ``` -------------------------------- ### Initialize Target Table (Rust) Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/merging-tables.md Initializes a target table with sample data using Rust. ```rust let url = "./data/simple_table".parse()?; let mut table = DeltaTable::create(url) .with_table_name("some-table") .with_save_mode(SaveMode::Overwrite) .with_columns( StructType::new(vec![ StructField::new( "x".to_string(), DataType::Primitive(PrimitiveType::Integer), true, ), StructField::new( "y".to_string(), DataType::Primitive(PrimitiveType::Integer), true, ), ]) .fields() .cloned(), ) .await?; let schema = Arc::new(Schema::new(vec![ Field::new("x", arrow::datatypes::DataType::Int32, true), Field::new("y", arrow::datatypes::DataType::Int32, true), ])); let mut record_batch_writer = deltalake::writer::RecordBatchWriter::for_table(&mut table)?; record_batch_writer .write(RecordBatch::try_new( schema.clone(), vec![ Arc::new(Int32Array::from(vec![1, 2, 3])), Arc::new(Int32Array::from(vec![4, 5, 6])), ], )?) .await?; record_batch_writer.flush_and_commit(&mut table).await?; ``` -------------------------------- ### Optimize operation output Source: https://github.com/delta-io/delta-rs/blob/main/docs/usage/optimize/small-file-compaction-with-optimize.md Example output showing the metrics returned after running the optimize operation. ```python {'numFilesAdded': 5, 'numFilesRemoved': 100, 'filesAdded': {'min': 39000, 'max': 238282, 'avg': 198425.6, 'totalFiles': 5, 'totalSize': 992128}, 'filesRemoved': {'min': 10244, 'max': 10244, 'avg': 10244.0, 'totalFiles': 100, 'totalSize': 1024400}, 'partitionsOptimized': 5, 'numBatches': 1, 'totalConsideredFiles': 100, 'totalFilesSkipped': 0, 'plannerStrategy': 'preserveLocality', 'preservedStableOrder': True, 'preserveInsertionOrder': True, 'maxBinSpanFiles': 20} ``` -------------------------------- ### Explain Query Plan for Non-Partitioned Column Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-daft.md Running explain on a query filtering by a non-partitioned column shows that all partitions must be scanned. ```text (...) == Optimized Logical Plan == * PythonScanOperator: DeltaLakeScanOperator(None) | File schema = first_name#Utf8, last_name#Utf8, country#Utf8, continent#Utf8 | Partitioning keys = [PartitionField(country#Utf8)] | Filter pushdown = col(continent) == lit("Asia") | Output schema = first_name#Utf8, last_name#Utf8, country#Utf8, continent#Utf8 == Physical Plan == * TabularScan: | Num Scan Tasks = 3 | Estimated Scan Bytes = 3045 | Clustering spec = { Num partitions = 3 } ``` -------------------------------- ### Define a Dagster Asset with Polars Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-dagster.md Example of defining a Dagster asset that returns a Polars DataFrame. ```python import polars as pl from dagster import asset @asset def iris_dataset() -> pl.DataFrame: return pl.read_csv( "https://docs.dagster.io/assets/iris.csv", new_columns=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], has_header=False ) ``` -------------------------------- ### Use the local filesystem OpenDAL service (fs://) Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/object-storage/opendal.md The fs service uses the bare fs:// scheme because it has no native counterpart. opendal.root scopes the operator at a directory, and the URL path is the table prefix within it. ```python storage_options = {"opendal.root": "/tmp/delta-data"} write_deltalake("fs://localhost/my_table", df, storage_options=storage_options) ``` -------------------------------- ### Defining a pandas DataFrame asset Source: https://github.com/delta-io/delta-rs/blob/main/docs/integrations/delta-lake-dagster.md Example of an asset that returns a pandas DataFrame, which can be read and written using the DeltaLakePandasIOManager. ```python import pandas as pd from dagster import asset @asset def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) ```