### Setup Virtual Environment for Benchmarking

Source: https://github.com/eventual-inc/daft/blob/main/benchmarking/parquet/README.md

Create and activate a virtual environment, then install benchmark dependencies. Ensure you are in the project's root directory.

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r benchmark-requirements.txt
```

--------------------------------

### Setup Development Environment

Source: https://github.com/eventual-inc/daft/blob/main/AGENTS.md

Initializes the Python virtual environment and installs necessary dependencies.

```bash
make .venv
```

--------------------------------

### Install Daft and Start Ray Cluster

Source: https://github.com/eventual-inc/daft/blob/main/docs/distributed/ray.md

Install Daft with Ray support and start a local Ray cluster. This is the first step for local setup.

```bash
pip install "daft[ray]"
ray start --head
```

--------------------------------

### Complete Daft and Ray Initialization Example

Source: https://github.com/eventual-inc/daft/blob/main/docs/observability/logging.md

Initialize Ray with advanced logging, environment variables, and a worker setup hook for debugging Daft. Configure logging before importing Daft.

```python
import logging
import ray
from ray.job_config import LoggingConfig
import daft

# A setup hook to configure logging on each worker
def configure_logging():
    from daft.logging import setup_logger
    # Example of setting Daft module to INFO level except for daft_distributed module
    setup_logger(level="INFO", exclude_prefix=["daft_distributed"])


configure_logging()

# Initialize Ray with advanced logging and environment settings
ray.init(
    dashboard_host="0.0.0.0",
    runtime_env={
        "env_vars": {
            "DAFT_PROGRESS_BAR": "0",
            "DAFT_DEBUG_DISPATCH": "1"
        },
        "worker_process_setup_hook": configure_logging,
    },
    logging_config=LoggingConfig(
        log_level="DEBUG",
    ),
    log_to_driver=True,
)
```

--------------------------------

### Install Daft and dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/udf-patterns.md

Install the required packages for the tutorial.

```bash
pip install daft aiohttp
```

--------------------------------

### Install Required Packages

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/window-functions.md

Install the necessary Python packages for the tutorial.

```bash
pip install daft pandas matplotlib
```

--------------------------------

### Install Daft, Pandas, and Matplotlib

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/window_functions/window_functions.ipynb

Installs the necessary Python packages for the tutorial. Ensure these are installed before running the code.

```python
!pip install daft pandas matplotlib
```

--------------------------------

### Install dependencies

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/text_to_image/text_to_image_generation.ipynb

Install the Daft library and necessary machine learning packages.

```bash
!pip install daft --pre --extra-index-url https://pypi.anaconda.org/daft-nightly/simple
!pip install transformers diffusers accelerate torch Pillow
```

--------------------------------

### Install Daft and Import

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/udf_patterns/udf_patterns.ipynb

Install the Daft library and import it for use. This is a prerequisite for all Daft operations.

```python
# pip install daft aiohttp
import daft
```

--------------------------------

### Initialize uv project and install dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/audio-transcription.md

Sets up a new project directory and installs Daft, SoundFile, and Whisper using uv.

```bash
mkdir transcribe_demo
cd transcribe_demo
# Initialize uv project
uv init
# Install dependencies
uv add daft soundfile openai-whisper
```

--------------------------------

### DESCRIBE SELECT Examples

Source: https://github.com/eventual-inc/daft/blob/main/docs/sql/statements/describe.md

Examples of retrieving execution plans for simple and complex SELECT queries.

```sql
DESCRIBE SELECT * FROM T;
```

```sql
DESCRIBE SELECT a.*, b.name FROM T a JOIN S b ON a.id = b.id;
```

--------------------------------

### Install Dependencies and Configure AWS

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/text-embeddings.md

Install the necessary Python packages and authenticate with AWS.

```bash
pip install "daft[ray]" turbopuffer torch sentence-transformers spacy accelerate transformers
python -m spacy download en_core_web_sm
```

```bash
aws sso login
```

--------------------------------

### Install Dependencies

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/talks_and_demos/iceberg_summit_2024.ipynb

Installs necessary packages for PyIceberg, Daft, and Ray integration.

```bash
!pip install 'pyiceberg[sql]'
!pip install 'daft[ray]' polars pandas
!pip install ray==2.20.0
!pip install sqlalchemy ipywidgets boto3 mypy_boto3_glue
```

--------------------------------

### Install Dependencies

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/minhash_dedupe/minhash_dedupe_common_crawl.ipynb

Install the necessary libraries for the deduplication pipeline.

```python
# Install additional dependencies
# For Google Colab or standalone environments:
%pip install 'daft[aws,pandas]' selectolax scipy matplotlib igraph ipywidgets

# For local development with uv (temporary installation):
# uv pip install 'daft[aws,pandas]' selectolax scipy matplotlib igraph ipywidgets
```

--------------------------------

### Install Daft with all dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/install.md

Installs Daft along with all available optional dependencies.

```bash
pip install -U "daft[all]"
```

--------------------------------

### Install dependencies using pip

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/audio-transcription.md

Installs Daft, SoundFile, and Whisper using pip.

```bash
pip install daft soundfile openai-whisper
```

--------------------------------

### Install Daft for Benchmarking

Source: https://github.com/eventual-inc/daft/blob/main/benchmarking/parquet/README.md

Install the Daft library into the activated virtual environment. This can be a released wheel or a local build.

```bash
pip install daft
```

--------------------------------

### Install Daft and Dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/image-generation.md

Installs the Daft library and other necessary packages for running the tutorial. Use --pre for the latest pre-release versions.

```bash
pip install daft --pre --extra-index-url https://pypi.anaconda.org/daft-nightly/simple
pip install transformers diffusers accelerate torch Pillow
```

--------------------------------

### Install Dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/modalities/images.md

Install the necessary libraries for PyTorch-based image classification.

```bash
pip install validators matplotlib Pillow torch torchvision
```

--------------------------------

### Install Rust and uv

Source: https://github.com/eventual-inc/daft/blob/main/docs/extensions/authoring.md

Commands to install the Rust toolchain and the uv package installer. These are prerequisites for setting up a Rust-based Daft extension project.

```bash
# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Install Daft with SQL support

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/sql.md

Install the necessary dependencies for SQL integration.

```bash
pip install -U "daft[sql]"
```

--------------------------------

### Install Daft

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/intro.ipynb

Use pip to install the Daft library.

```python
!pip install daft
```

--------------------------------

### Install Daft with Unity support

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/unity_catalog.md

Required installation command to enable Unity Catalog functionality.

```bash
pip install daft[unity]
```

--------------------------------

### Install Daft via pip

Source: https://github.com/eventual-inc/daft/blob/main/docs/install.md

Standard installation command for the Daft library.

```bash
pip install -U daft
```

--------------------------------

### Install Daft and dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/common-crawl-daft-tutorial.md

Install the necessary packages including Daft with transformers support and spacy.

```bash
!pip install uv && uv pip install "daft[transformers]" spacy
```

--------------------------------

### Install Daft and Dependencies

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/document_processing/document_processing_tutorial.ipynb

Install the necessary libraries for PDF processing, OCR, and machine learning tasks.

```bash
! pip install daft pillow pydantic PyMuPDF pytesseract sentence-transformers pydantic-to-pyarrow pdf2image accelerate
```

--------------------------------

### Install Daft and dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/document-processing.md

Install the necessary libraries for PDF processing, AWS integration, and machine learning tasks.

```bash
pip install "daft[aws]" pillow pydantic PyMuPDF pytesseract sentence-transformers pydantic-to-pyarrow pdf2image accelerate
```

--------------------------------

### Install Daft and dependencies

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/image_querying/top_n_red_color.ipynb

Install the necessary libraries for Daft with AWS and Ray support, along with image processing tools.

```bash
!pip install daft[aws,ray]
!pip install Pillow numpy
```

--------------------------------

### Install Daft with Kafka support

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/kafka.md

Install the necessary dependencies to enable Kafka connectivity in Daft.

```bash
pip install -U "daft[kafka]"
```

```bash
pip install confluent-kafka
```

--------------------------------

### Install Daft and Paimon dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/paimon.md

Install the required packages for Daft and Paimon integration.

```bash
pip install daft
pip install pypaimon
```

```bash
pip install "daft[aws]"
```

--------------------------------

### Install Daft with AWS support and Sentence Transformers

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/llms-red-pajamas.md

Install the Daft library with AWS support and the sentence-transformers library for embedding computation. This is a prerequisite for running the tutorial.

```bash
pip install "daft[aws]" sentence-transformers accelerate
```

--------------------------------

### Install Daft and dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/querying-images.md

Install the necessary libraries for Daft, including AWS and Ray support, along with Pillow and NumPy.

```bash
pip install "daft[aws,ray]"
pip install Pillow numpy
```

--------------------------------

### DESCRIBE TABLE Examples

Source: https://github.com/eventual-inc/daft/blob/main/docs/sql/statements/describe.md

Examples of retrieving table schema information using various identifier formats.

```sql
DESCRIBE T;
```

```sql
DESCRIBE TABLE T;
```

```sql
DESCRIBE my_catalog.my_namespace.my_table;
```

--------------------------------

### Install Daft and dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/notebooks/quickstart.ipynb

Install the Daft library with OpenAI extras and additional image processing libraries.

```bash
!pip install -U "daft[openai]"  # Includes OpenAI extras needed for this quickstart
```

```bash
!pip install numpy pillow
```

--------------------------------

### Install and Run Extension Tests

Source: https://github.com/eventual-inc/daft/blob/main/examples/hello/README.md

Install the extension in the project's virtual environment and run the tests using pytest.

```bash
# Install the extension in the project .venv
uv pip install -e .

# Run the tests!
pytest -v
```

--------------------------------

### Install Daft and Deltalake

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/talks_and_demos/data-ai-summit-2024.ipynb

Install the Daft library and a compatible version of deltalake for data lake operations.

```python
!pip install daft deltalake<0.17
```

--------------------------------

### Install spaCy and Download Model

Source: https://github.com/eventual-inc/daft/blob/main/docs/modalities/text.md

Install the spaCy library and download a language model for sentence chunking. This is a prerequisite for the Python UDF.

```bash
pip install -U spacy
python -m spacy download en_core_web_sm
```

--------------------------------

### Install Daft and Dependencies

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/mnist.ipynb

Installs the Daft library and necessary dependencies like Pillow, Torch, and Torchvision. Run these commands in your environment before proceeding.

```python
%pip install daft
%pip install Pillow torch torchvision
```

--------------------------------

### Install Daft with Ray Support

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/text_to_image/using_cloud_with_ray.ipynb

Install the Daft library with the necessary Ray dependencies and the Pillow library for image processing.

```python
!pip install daft[ray]
!pip install Pillow
```

--------------------------------

### Install Daft with Delta Lake support

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/delta_lake.md

Install the necessary dependencies to enable Delta Lake functionality in Daft.

```bash
pip install -U "daft[deltalake]"
```

--------------------------------

### Docstring Template Example

Source: https://github.com/eventual-inc/daft/blob/main/docs/contributing/development.md

Follows Google style Python docstrings. Includes sections for summary, arguments, returns, raises, warnings, notes, and runnable examples.

```python
def method(args) -> return:
    """Summary of method

    Args:
        arg1: description

    Returns:
        return1: description

    Raises: title (optional)
         description

    Warning: title (optional)
         description

    Note: title (optional)
         description

    Examples: (make sure this is plural!)
        >>> code example (this needs to be a runnable `doctest` example)
        output

    Tip: title (optional)
         description
    """
```

--------------------------------

### Python Setup Script for Rust Extension

Source: https://github.com/eventual-inc/daft/blob/main/docs/extensions/authoring.md

A Python setup script that configures setuptools-rust to compile a cdylib for Daft extensions. It specifies the target library name and path, ensuring the compiled extension is placed correctly within the Python package.

```python
from setuptools import find_packages, setup
from setuptools_rust import Binding, RustExtension

setup(
    packages=find_packages(),
    rust_extensions=[
        RustExtension(
            "hello.libhello",   # <python_package>.<lib_name>
            path="Cargo.toml",
            binding=Binding.NoBinding,
            strip=True,
        )
    ],
)
```

--------------------------------

### Guided Choice (Classification) with Daft and vLLM

Source: https://github.com/eventual-inc/daft/blob/main/docs/ai-functions/prompt.md

Use Daft's `prompt` function with `extra_body` to perform guided choice classification. This example classifies sentiment by constraining the output to predefined choices.

```python
df = daft.from_pydict({
    "statement": [
        "vLLM is wonderful",
        "Daft is wicked fast",
        "Slow inference sucks",
    ]
})

df = df.with_column(
    "sentiment",
    prompt(
        messages=format("Classify this sentiment: {}", daft.col("statement")),
        model="google/gemma-3-4b-it",
        extra_body={"structured_outputs": {"choice": ["positive", "negative"]}},
    )
)
```

--------------------------------

### Run Example 1

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/audio-transcription.md

Executes the Python script for transcribing audio bytes from a dataset using uv.

```bash
uv run example_1.py
```

--------------------------------

### IoT Data Storage Example

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/bigtable.md

Prepare sensor data by creating a composite row key and write it to Bigtable. This example demonstrates reading from Parquet and transforming data before writing.

```python
import daft
from daft import col

# Read sensor data
df = daft.read_parquet("s3://bucket/sensors/*.parquet")

# Prepare for Bigtable (create composite row key)
df = df.with_column(
    "row_key",
    col("device_id").str.concat("#").str.concat(col("timestamp").cast(str)),
)

# Write to Bigtable
df.write_bigtable(
    project_id="iot-project",
    instance_id="sensor-data",
    table_id="readings",
    row_key_column="row_key",
    column_family_mappings={
        "temperature": "metrics",
        "humidity": "metrics",
        "device_id": "metadata",
        "timestamp": "metadata",
    },
)
```

--------------------------------

### Implement a Local File DataSink in Python

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/custom.md

Create a custom sink by inheriting from DataSink and implementing the start, write, and finalize methods. This example uses a local file writer to process micropartitions.

```python
import uuid
from collections.abc import Iterator
from pathlib import Path
from typing import Any

from daft.datatype import DataType
from daft.io import DataSink
from daft.io.sink import WriteResult
from daft.recordbatch import MicroPartition
from daft.schema import Schema


class LocalFileDataSink(DataSink[dict]):
    """A simple data sink that writes data to local files."""

    def __init__(
        self,
        output_dir: str | Path,
        filename_prefix: str = "data",
        max_rows_per_file: int = 10
    ):
        """Initialize the local file data sink.

        Args:
            output_dir: Directory where files will be written
            filename_prefix: Prefix for generated filenames
            max_rows_per_file: Maximum rows to write per file
        """
        self.output_dir = Path(output_dir)
        self.filename_prefix = filename_prefix
        self.max_rows_per_file = max_rows_per_file

        self.output_dir.mkdir(parents=True, exist_ok=True)

        self._result_schema = Schema._from_field_name_and_types([
            ("files_written", DataType.int64()),
            ("total_rows", DataType.int64()),
            ("total_bytes", DataType.int64()),
            ("output_directory", DataType.string()),
        ])

    def name(self) -> str:
        """Return a descriptive name for this sink."""
        return "Local File Data Sink"

    def schema(self) -> Schema:
        """Return the schema for the results of finalize()."""
        return self._result_schema

    def start(self) -> None:
        """Called once at the beginning of the write process.

        This is a good place to initialize resources, create directories,
        start a transaction, etc.
        """
        print(f"Starting write to {self.output_dir}")
        print(f"Max rows per file: {self.max_rows_per_file}")

    def write(self, micropartitions: Iterator[MicroPartition]) -> Iterator[WriteResult[dict]]:
        """Process each micropartition and write to local files.

        This method is called for each micropartition and should yield
        WriteResults. When run in a distributed setting, this method is called
        in parallel on each worker.

        Args:
            micropartitions: Iterator of micropartitions to process

        Yields:
            WriteResult: Information about each write operation
        """
        for micropartition in micropartitions:
            # Convert micropartition to a format we can write.
            data = self._prepare_data_for_writing(micropartition)

            # Write data to files (potentially multiple files if data is large).
            # Split data into chunks based on max_rows_per_file.
            for i in range(0, len(data), self.max_rows_per_file):
                chunk = data[i:i + self.max_rows_per_file]
                write_result = self._write_data_to_files(chunk)
                 # Yield results for each file written.
                yield WriteResult(
                    result=write_result,
                    bytes_written=write_result["bytes_written"],
                    rows_written=write_result["rows_written"]
                )

    def finalize(self, write_results: list[WriteResult[dict]]) -> MicroPartition:
        """Aggregate all write results into a final summary.

        This method is called after all writes complete and should
        return a single MicroPartition with summary information.

        This is a good place to commit a transaction, clean up resources, etc.

        Args:
            write_results: List of all WriteResult objects from write()

        Returns:
            MicroPartition: Summary of all write operations
        """
        if not write_results:
            return MicroPartition.empty(self._result_schema)

        total_files = len(write_results)
        total_rows = sum(wr.rows_written for wr in write_results)
        total_bytes = sum(wr.bytes_written for wr in write_results)

        print(f"Write completed: {total_files} files, {total_rows} rows, {total_bytes} bytes")

        return MicroPartition.from_pydict({
            "files_written": [total_files],
            "total_rows": [total_rows],
            "total_bytes": [total_bytes],
            "output_directory": [str(self.output_dir)],
        })

    def _prepare_data_for_writing(self, micropartition: MicroPartition) -> list[dict]:
        """Convert a micropartition to a list of dictionaries for writing.

        Args:
            micropartition: The data to prepare

        Returns:
            List of dictionaries representing the data
        """
        return micropartition.to_pylist()

    def _write_data_to_files(self, data: list[Any]) -> dict:
```

--------------------------------

### Install uv

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/audio-transcription.md

Installs the uv package manager. Ensure you have curl installed.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Display setup.py

Source: https://github.com/eventual-inc/daft/blob/main/docs/extensions/authoring.md

Displays the contents of the setup.py file, which is used by setuptools to build and package the Python project, including the Rust extension.

```bash
cat setup.py
```

--------------------------------

### Quickstart Template

Source: https://github.com/eventual-inc/daft/blob/main/docs/datasets/common-crawl.md

Template for initializing a Common Crawl dataset load with explicit in_aws parameter.

```python
import daft

# If you are running this code locally, set `in_aws = True`. This will use S3.
# Otherwise, set `in_aws = False`. This will use HTTPS URLs for the files.
# You must **explicitly** set the `in_aws` parameter.
in_aws: bool = ...
```

--------------------------------

### Initialize development environment

Source: https://github.com/eventual-inc/daft/blob/main/docs/contributing/development.md

Commands to prepare the local development environment and install pre-commit hooks.

```bash
make .venv
```

```bash
make hooks
```

--------------------------------

### Install daft-h3 extension

Source: https://github.com/eventual-inc/daft/blob/main/docs/extensions/community.md

Install the daft-h3 package via pip.

```bash
pip install daft-h3
```

--------------------------------

### Install Daft and Sentence Transformers

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/embeddings/daft_tutorial_embeddings_stackexchange.ipynb

Installs the necessary Python packages for Daft with AWS support and sentence-transformers for embedding computation. Accelerate is also installed for optimized model performance.

```python
# Install Daft!
!pip install 'daft[aws]'

# We will use sentence-transformers for computing embeddings.
!pip install sentence-transformers accelerate
```

--------------------------------

### Create a local SQLite table

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/sql.md

Setup a sample SQLite database for testing Daft SQL reads.

```python
import sqlite3

connection = sqlite3.connect("example.db")
connection.execute(
    "CREATE TABLE IF NOT EXISTS books (title TEXT, author TEXT, year INTEGER)"
)
connection.execute(
    """
INSERT INTO books (title, author, year)
VALUES
    ('The Great Gatsby', 'F. Scott Fitzgerald', 1925),
    ('To Kill a Mockingbird', 'Harper Lee', 1960),
    ('1984', 'George Orwell', 1949),
    ('The Catcher in the Rye', 'J.D. Salinger', 1951)
"""
)
connection.commit()
connection.close()
```

--------------------------------

### Initialize Daft and Ray Environment

Source: https://github.com/eventual-inc/daft/blob/main/benchmarking/vllm/generate_data.ipynb

Sets up the Ray runner for Daft and imports necessary utilities for dataset generation.

```python
from vllm.benchmarks.datasets import PrefixRepetitionRandomDataset
from vllm.transformers_utils.tokenizer import get_tokenizer
import daft
from daft.functions import monotonically_increasing_id
import ray
ray.init()
daft.set_runner_ray()
```

--------------------------------

### Install Daft dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/minhash-dedupe.md

Install the necessary libraries for running the deduplication pipeline.

```bash
!pip install 'daft[aws,pandas]' selectolax scipy matplotlib igraph
```

--------------------------------

### View SQL Pushdown Output

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/sql.md

Example output showing the physical plan with the injected SQL query containing filters and projections.

```text
..
== Physical Plan ==
..
|   SQL Query = SELECT refresh_date, term, rank FROM
 (SELECT * FROM `bigquery-public-data.google_trends.top_terms`)
 AS subquery WHERE rank = 1 AND refresh_date >= CAST('2024-04-01' AS DATE)
 AND refresh_date < CAST('2024-04-08' AS DATE)
```

--------------------------------

### Setup Iceberg Catalog for Daft

Source: https://github.com/eventual-inc/daft/blob/main/docs/configuration/sessions-usage.md

Shows how to set up an SQLite-backed Iceberg catalog for use with Daft. Requires `pyiceberg[sql-sqlite]` and creating a directory for the catalog.

```python
from daft import Catalog
from pyiceberg.catalog.sql import SqlCatalog

# don't forget to `mkdir -p /tmp/daft/example`
tmpdir = "/tmp/daft/example"

# create a pyiceberg catalog backed by sqlite
iceberg_catalog = SqlCatalog(
    "default",
    **{
        "uri": f"sqlite:///{tmpdir}/catalog.db",
        "warehouse": f"file://{tmpdir}",
    },
)

# creating a daft catalog from the pyiceberg catalog implementation
catalog = Catalog.from_iceberg(iceberg_catalog)

# check
catalog.name

```

--------------------------------

### Execute SQL with Sessions

Source: https://github.com/eventual-inc/daft/blob/main/docs/sql/index.md

Demonstrates creating a session and executing SQL queries on temporary tables.

```python
import daft

from daft import Session

# create a session
sess = Session()

# create temp tables
sess.create_temp_table("T", daft.from_pydict({ "a": [ 0, 1 ] }))
sess.create_temp_table("S", daft.from_pydict({ "b": [ 1, 0 ] }))

# execute sql
sess.sql("SELECT * FROM T, S").show()
```

--------------------------------

### Install Transformers dependency

Source: https://github.com/eventual-inc/daft/blob/main/docs/modalities/text.md

Install the optional dependency required for using the transformers provider.

```bash
pip install -U "daft[transformers]"
```

--------------------------------

### Install ClickHouse dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/clickhouse.md

Install the required clickhouse-connect package to enable ClickHouse support.

```bash
pip install clickhouse-connect
```

--------------------------------

### Import required libraries

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/common-crawl-daft-tutorial.md

Standard imports and environment configuration for the tutorial.

```python
import os

# We need to do this _before_ importing torch
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

import json
from collections.abc import Iterator
from datetime import datetime
from typing import TypedDict

import spacy
import spacy.cli
import torch
from transformers import AutoConfig

import daft
from daft import col
```

--------------------------------

### Install Bigtable Dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/bigtable.md

Install the necessary package for Bigtable support using pip.

```bash
pip install google-cloud-bigtable
```

--------------------------------

### Initialize IO configuration

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/talks_and_demos/linkedin-03-05-2024.ipynb

Set up an anonymous IO configuration for accessing public cloud storage.

```python
import daft

ANONYMOUS_IO_CONFIG = daft.io.IOConfig(s3=daft.io.S3Config(anonymous=True, region_name="us-west-2"))
```

--------------------------------

### Install OpenAI dependency

Source: https://github.com/eventual-inc/daft/blob/main/docs/modalities/text.md

Install the optional dependency required for using OpenAI or OpenAI-compatible APIs.

```bash
pip install -U "daft[openai]"
```

--------------------------------

### Enable Debug Logging with setup_logger()

Source: https://github.com/eventual-inc/daft/blob/main/docs/observability/logging.md

Quickly enable debug logging for Daft using the `setup_logger()` utility function. This function simplifies the process of configuring Daft's logging output.

```python
# Import daft first to access the utility function
import daft
from daft.logging import setup_logger

# Configure debug logging
setup_logger()
```

--------------------------------

### Set Up Environment Variables and Configuration

Source: https://github.com/eventual-inc/daft/blob/main/docs/examples/mm_structured_outputs.md

Load environment variables from a .env file and configure model ID, data limit, and HuggingFace API credentials for Daft.

```python
import os
from dotenv import load_dotenv

load_dotenv()

# Configuration
MODEL_ID = "Qwen/Qwen3-VL-8B-Instruct"
LIMIT = 50  # Keep low for interactive demo

# HuggingFace Inference Provider (hosted Qwen3-VL endpoints)
OPENAI_API_KEY = os.getenv("HF_TOKEN")
OPENAI_BASE_URL = "https://router.huggingface.co/v1"
```

--------------------------------

### Launch vLLM OpenAI Compatible Server

Source: https://github.com/eventual-inc/daft/blob/main/docs/ai-functions/prompt.md

Run this command in your terminal to start a vLLM server compatible with the OpenAI API. Ensure you have an L4 GPU instance or larger. The server may take several minutes to become ready.

```bash
python -m vllm.entrypoints.openai.api_server \
  --model google/gemma-3-4b-it \
  --guided-decoding-backend guidance \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.85 \
  --host 0.0.0.0 --port 8000
```

--------------------------------

### Set Catalog and Namespace with USE

Source: https://github.com/eventual-inc/daft/blob/main/docs/sql/statements/use.md

Use these statements to define the session context. The first sets only the catalog, while the second sets both the catalog and namespace.

```sql
USE my_catalog;
```

```sql
USE my_catalog.my_namespace;
```

--------------------------------

### Initialize an End-to-End Document Pipeline

Source: https://github.com/eventual-inc/daft/blob/main/docs/modalities/embeddings.md

Sets up the execution configuration and provider credentials for a pipeline involving LLM metadata extraction and vector generation.

```python
# /// script
# description = "This example shows how using LLMs and embedding models, Daft chunks documents, extracts metadata, generates vectors, and writes them to any vector database..."
# dependencies = ["daft[openai, turbopuffer]", "pymupdf"]
# ///
import os
import daft
from daft import col, lit
from daft.functions import embed_text, prompt, file, unnest, monotonically_increasing_id
from pydantic import BaseModel

class Classifier(BaseModel):
    title: str
    author: str
    year: int
    keywords: list[str]
    abstract: str

daft.set_execution_config(enable_dynamic_batching=True)
daft.set_provider("openai", api_key=os.environ.get("OPENAI_API_KEY"))
```

--------------------------------

### Install Daft and dependencies

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/talks_and_demos/linkedin-03-05-2024.ipynb

Use pip to install Daft with support for specific table formats and ipywidgets.

```python
!pip install -U 'daft[iceberg,hudi,deltalake]'
!pip install -U ipywidgets
```

--------------------------------

### Install Daft with Gravitino support

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/gravitino.md

Install Daft with the `gravitino` option to enable Gravitino connector functionality.

```bash
pip install daft[gravitino]
```

--------------------------------

### Install MCAP dependencies

Source: https://github.com/eventual-inc/daft/blob/main/docs/connectors/mcap.md

Install the required mcap package to enable MCAP file support in Daft.

```bash
pip install mcap
```

--------------------------------

### Python Project Setup with setuptools-rust

Source: https://github.com/eventual-inc/daft/blob/main/docs/extensions/authoring.md

Configuration for a Python project to build a Rust extension using setuptools-rust. Specifies build requirements and project metadata.

```toml
[build-system]
requires = ["setuptools", "setuptools-rust"]
build-backend = "setuptools.build_meta"

[project]
name = "hello"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["daft"]

[project.optional-dependencies]
test = ["pytest"]
```

--------------------------------

### Connect to SQL Catalog

Source: https://github.com/eventual-inc/daft/blob/main/tutorials/talks_and_demos/iceberg_summit_2024.ipynb

Initializes a local SQLite-backed Iceberg catalog.

```python
from pyiceberg.catalog.sql import SqlCatalog

warehouse_path = "/tmp/warehouse"
catalog = SqlCatalog(
    "default",
    **{
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)
```