### Install Sail and PySpark

Source: https://github.com/lakehq/sail/blob/main/README.md

Install the Sail Python package and PySpark client for basic usage. This is the recommended first step for getting started.

```bash
pip install pysail
pip install "pyspark-client"
```

--------------------------------

### Start Docker Compose Services

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/object-store.md

Run this command to start all defined services in the background.

```bash
docker compose up -d
```

--------------------------------

### Install Rust Toolchains

Source: https://github.com/lakehq/sail/blob/main/docs/development/setup/prerequisites.md

Install the stable and nightly Rust toolchains, including the 'llvm-tools-preview' component for the stable toolchain, using rustup.

```bash
rustup toolchain install stable --profile default --component llvm-tools-preview
rustup toolchain install nightly --profile default
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/lakehq/sail/blob/main/docs/development/documentation/index.md

Installs all necessary project dependencies using pnpm. This is a prerequisite for building the documentation and formatting the code.

```bash
pnpm install
```

--------------------------------

### Install Built Python Package

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/python.md

Installs the pre-built Python package into the default Hatch environment.

```bash
hatch run install-pysail
```

--------------------------------

### Install LLVM on Windows

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/rust.md

Install LLVM using Scoop on Windows. This is a prerequisite for setting up the LLVM linker for Rust builds on Windows.

```bash
scoop install llvm
```

--------------------------------

### Install Ibis with PySpark Backend

Source: https://github.com/lakehq/sail/blob/main/docs/guide/integrations/ibis.md

Install Ibis with the PySpark backend and a Spark Connect client using pip.

```bash
pip install "ibis-framework[pyspark]>=12,<13"
```

--------------------------------

### Use Sail Library to Start Spark Connect Server

Source: https://github.com/lakehq/sail/blob/main/docs/introduction/getting-started/index.md

Programmatically start a Spark Connect server using the Sail library and connect to it with PySpark. This example demonstrates running SQL queries and reading multimodal data.

```python
from pysail.spark import SparkConnectServer
from pyspark.sql import SparkSession

server = SparkConnectServer()
server.start()
_, port = server.listening_address

spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate()

# Run a simple query
spark.sql("SELECT 1 + 1").show()

# Use the DataFrame API to read multimodal data
spark.read.format("binaryFile").option("pathGlobFilter", "*.png").load("/path/to/data").show()

# Use Spark SQL to create a table that refers to multimodal data in an object storage
spark.sql("""
CREATE TABLE pdfs
USING binaryFile
OPTIONS (pathGlobFilter '*.pdf')
LOCATION 's3://my-bucket/path/to/data'
""")
spark.sql("SELECT * FROM pdfs").show()
```

--------------------------------

### Install Package in Test Environments

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/python.md

Installs the Python package into all 'test' matrix environments, preparing for multi-version Spark testing.

```bash
hatch run test:install-pysail
```

--------------------------------

### Batch Reader Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/python/index.md

Demonstrates how to implement a batch reader for a custom Python data source.

```python-console
from pyspark.sql import SparkSession
from pyspark.sql.datasource import DataSource

class CustomReader(DataSource):
    def __init__(self, sparkSession, options):
        super().__init__(sparkSession, options)

    def schema(self):
        return "name STRING, age INT"

    def read(self):
        data = [("Alice", 1), ("Bob", 2)]
        return self.spark.createDataFrame(data, self.schema())

spark = SparkSession.builder.appName("PythonDataSourceExample").getOrCreate()

# Register the custom data source
spark.sparkSession.register_data_source("custom_reader", CustomReader)

# Use the custom data source
df = spark.read.format("custom_reader").load()
df.show()

# Expected output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice|  1|
# |  Bob|  2|
# +-----+---+

```

--------------------------------

### Install Rustup and Node.js via Homebrew

Source: https://github.com/lakehq/sail/blob/main/docs/development/setup/prerequisites.md

Use Homebrew to install the Rust toolchain manager and the Node.js runtime on macOS.

```bash
brew install rustup
brew install node
```

--------------------------------

### Install dbt-sail

Source: https://github.com/lakehq/sail/blob/main/docs/guide/integrations/dbt.md

Install the dbt-sail adapter using pip. This command also installs necessary dependencies like dbt-spark and pysail.

```bash
pip install dbt-sail
```

--------------------------------

### Start Sail Server via CLI

Source: https://github.com/lakehq/sail/blob/main/README.md

Start a local Sail server instance using the command-line interface. Ensure the port is accessible for client connections.

```bash
sail spark server --port 50051
```

--------------------------------

### Install Cross-Compilation Target

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/cross-compilation.md

Installs the specified Rust target toolchain for cross-compilation.

```bash
rustup target add aarch64-unknown-linux-gnu
```

--------------------------------

### Install OpenAPI Generator CLI

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/iceberg-openapi-generation.md

Install the OpenAPI Generator CLI using Homebrew. This is a prerequisite for regenerating the REST client.

```bash
brew install openapi-generator
```

--------------------------------

### Install pysail with JDBC extra

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/jdbc/index.md

Install the pysail package with the 'jdbc' extra to enable the JDBC data source functionality.

```bash
pip install pysail[jdbc]
```

--------------------------------

### Dockerfile for Sail Quick Start

Source: https://github.com/lakehq/sail/blob/main/docs/guide/deployment/docker-images/quick-start.md

Create this Dockerfile in an empty directory to build the Sail Docker image. It uses the Sail Python package to set up the image.

```docker
# Use the official Python runtime as a parent image
FROM python:3.10-slim

# Set the working directory in the container
WORKDIR /app

# Install the Sail Python package
RUN pip install --no-cache-dir "sail>=0.1.0"

# Expose the default port
EXPOSE 8000

# Define environment variable
ENV NAME World

# Run sail in the container
CMD ["sail", "--host", "0.0.0.0", "--port", "8000"]
```

--------------------------------

### Authenticate with Kerberos and Start Sail Server

Source: https://github.com/lakehq/sail/blob/main/docs/guide/catalog/hms.md

This Python snippet demonstrates how to authenticate with Kerberos using `kinit` and then start the Sail Spark Connect server. Ensure you have a valid keytab and Kerberos ticket cache.

```python
import subprocess
from pysail.spark import SparkConnectServer

# authenticate with Kerberos
subprocess.run([
    "kinit", "-kt",
    "/path/to/user.keytab",
    "username@YOUR.REALM"
], check=True)

# start the Sail server
server = SparkConnectServer(ip="0.0.0.0", port=50051)
server.start(background=False)
```

--------------------------------

### Install Development Tools via Homebrew

Source: https://github.com/lakehq/sail/blob/main/docs/development/setup/prerequisites.md

Install essential development tools such as Protocol Buffers, Hatch, Maturin, Zig, and pnpm using Homebrew on macOS.

```bash
brew install protobuf hatch maturin zig pnpm
```

--------------------------------

### Install Sail from PyPI

Source: https://github.com/lakehq/sail/blob/main/docs/introduction/installation/index.md

Use this command to install the Sail Python package from PyPI. Ensure you replace `{{ libVersion }}` with the desired version.

```bash-vue
pip install "pysail=={{ libVersion }}"
```

--------------------------------

### Minute Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a minute interval literal.

```sql
INTERVAL '5000' MINUTE
```

--------------------------------

### Day Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a day interval literal.

```sql
INTERVAL -'365' DAY
```

--------------------------------

### Install Azure Storage Python Client Libraries

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/object-store.md

Installs the necessary Python libraries for interacting with Azure Blob Storage and DataLake Storage.

```bash
pip install azure-storage-blob azure-storage-file-datalake
```

--------------------------------

### Install Package in Specific Test Environment

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/python.md

Installs the Python package into a specific 'test' environment, for example, for Spark version 3.5.7.

```bash
hatch run test.spark-3.5.7:install-pysail
```

--------------------------------

### Install pysail with Vortex extra

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/vortex/index.md

Install the pysail package with the vortex extra to use the Vortex data source. This also installs the vortex-data Python library.

```bash
pip install pysail[vortex]
```

--------------------------------

### Day to Minute Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a day to minute interval literal.

```sql
INTERVAL '100 10:30' DAY TO MINUTE
```

--------------------------------

### Install PySail and PySpark Client for Spark 4.1

Source: https://github.com/lakehq/sail/blob/main/docs/introduction/getting-started/index.md

Install the necessary packages for using Sail with the Spark 4.1 client.

```bash
pip install "pysail=={{ libVersion }}"
pip install "pyspark-client==4.1.1"
```

--------------------------------

### Build Python API Docs from Current Release

Source: https://github.com/lakehq/sail/blob/main/docs/reference/python.md

Use these commands to build the Python API documentation from the current release. Ensure hatch is installed and the 'docs' environment is created.

```bash
hatch env create docs
hatch run docs:pip install pysail
hatch run docs:build
```

--------------------------------

### Start Spark Connect Server

Source: https://github.com/lakehq/sail/blob/main/docs/guide/cli/index.md

Launch a Spark Connect server using Sail for computation, specifying the IP address and port.

```bash
sail spark server --ip 127.0.0.1 --port 50051
```

--------------------------------

### Start Spark Connect Server for Direct MCP Server Execution

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/mcp-server.md

Starts a Spark Connect server, which is a prerequisite for running the MCP server Python script directly. This is part of a workaround to avoid slow binary rebuilds.

```bash
scripts/spark-tests/run-server.sh
```

--------------------------------

### Month Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a month interval literal.

```sql
INTERVAL '24' MONTH
```

--------------------------------

### Run VitePress Development Server

Source: https://github.com/lakehq/sail/blob/main/docs/development/documentation/index.md

Starts the VitePress development server for live preview. Changes to Markdown files are automatically reflected in the browser without manual refresh.

```bash
pnpm run docs:dev
```

--------------------------------

### Start PySpark Shell

Source: https://github.com/lakehq/sail/blob/main/docs/guide/cli/index.md

Initiate an interactive PySpark shell session powered by Sail.

```bash
sail spark shell
```

--------------------------------

### Day to Hour Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a day to hour interval literal.

```sql
INTERVAL '-10 05' DAY TO HOUR
```

--------------------------------

### Batch Arrow Reader Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/python/index.md

Demonstrates how to implement a batch reader for a custom Python data source using Apache Arrow for zero-copy data exchange.

```python-console
from pyspark.sql import SparkSession
from pyspark.sql.datasource import DataSource
import pyarrow as pa

class CustomArrowReader(DataSource):
    def __init__(self, sparkSession, options):
        super().__init__(sparkSession, options)

    def schema(self):
        return "name STRING, age INT"

    def read(self):
        # Using pyarrow to create data
        names = pa.array(['Alice', 'Bob'], type=pa.string())
        ages = pa.array([1, 2], type=pa.int64())
        table = pa.Table.from_arrays([names, ages], names=['name', 'age'])
        return self.spark.createDataFrame(table, self.schema())

spark = SparkSession.builder.appName("PythonArrowDataSourceExample").getOrCreate()

# Register the custom data source
spark.sparkSession.register_data_source("custom_arrow_reader", CustomArrowReader)

# Use the custom data source
df = spark.read.format("custom_arrow_reader").load()
df.show()

# Expected output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice|  1|
# |  Bob|  2|
# +-----+---+

```

--------------------------------

### Day to Second Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a day to second interval literal, including fractional seconds.

```sql
INTERVAL '100 10:30:40.999999' DAY TO SECOND
```

--------------------------------

### Minute to Second Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a minute to second interval literal, including fractional seconds.

```sql
INTERVAL '-'-'5000':'30.5' MINUTE TO SECOND
```

--------------------------------

### Create Regular View (Error Example)

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/catalog/test_temp_view.txt

Attempts to create a regular view named 't3' in the 'global_temp' database, which is not allowed and results in an AnalysisException.

```python
>>> spark.sql("CREATE VIEW global_temp.t3 AS SELECT 1")
Traceback (most recent call last):
...pyspark.errors.exceptions.connect.AnalysisException: ...
```

--------------------------------

### Configure Memory Catalog with Environment Variable

Source: https://github.com/lakehq/sail/blob/main/docs/guide/catalog/memory.md

This example shows how to configure the memory catalog provider using the SAIL_CATALOG__LIST environment variable. It sets up a catalog named 'sail' with an initial database 'dev.analytics' and a comment.

```bash
export SAIL_CATALOG__LIST='[{type="memory", name="sail", initial_database=["dev", "analytics"], initial_database_comment="Development analytics database"}]'
```

--------------------------------

### Start Sail Server via Python API

Source: https://github.com/lakehq/sail/blob/main/README.md

Start a local Sail server programmatically using the Python API. The server can be run in the background or foreground.

```python
from pysail.spark import SparkConnectServer

server = SparkConnectServer(port=50051)
server.start(background=False)
```

--------------------------------

### Hour Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for an hour interval literal.

```sql
INTERVAL '123' HOUR
```

--------------------------------

### Install MCP Dependencies

Source: https://github.com/lakehq/sail/blob/main/docs/guide/integrations/mcp-server.md

Install the PySail library with MCP dependencies and the PySpark client. Ensure you replace `{{ libVersion }}` with the correct library version.

```bash-vue
pip install "pysail[mcp]=={{ libVersion }}" "pyspark-client==4.1.1"
```

--------------------------------

### GROUP BY ROLLUP with GROUPING and Alternative ORDER BY

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/test_order_by_aggregate.txt

Similar to the previous example, but demonstrates an alternative syntax for ordering by the GROUPING function result.

```sql
SELECT brand, COUNT(*), GROUPING(brand) AS g
FROM cars
GROUP BY ROLLUP (brand)
ORDER BY GROUPING(brand), brand
```

--------------------------------

### Hour to Minute Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for an hour to minute interval literal.

```sql
INTERVAL -'-15:45' HOUR TO MINUTE
```

--------------------------------

### Build Python Package Without Installation

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/python.md

Builds the Python package, making the output available in the target/wheels directory without installing it.

```bash
hatch run maturin build
```

--------------------------------

### Install PySail and PySpark Connect for Spark 3.5

Source: https://github.com/lakehq/sail/blob/main/docs/introduction/getting-started/index.md

Install Sail and the full PySpark package with Spark Connect dependencies for Spark version 3.5.

```bash
pip install "pysail=={{ libVersion }}"
pip install "pyspark[connect]==3.5.7
```

--------------------------------

### Build Python API Docs from Local Build

Source: https://github.com/lakehq/sail/blob/main/docs/reference/python.md

Use these commands to build the Python API documentation from a local build. This requires building the full project first and then installing pysail within the 'docs' environment.

```bash
hatch env create docs
hatch run docs:install-pysail
hatch run docs:build
```

--------------------------------

### Build and Install Python Package for Development

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/python.md

Builds and installs the Python package in an editable format within the default Hatch environment. Changes to Python code are reflected immediately, but Rust code changes require rerunning this command.

```bash
hatch run maturin develop
```

--------------------------------

### Install PySail and PySpark Connect for Spark 4.1

Source: https://github.com/lakehq/sail/blob/main/docs/introduction/getting-started/index.md

Install Sail and the full PySpark package with Spark Connect dependencies for Spark version 4.1.

```bash
pip install "pysail=={{ libVersion }}"
pip install "pyspark[connect]==4.1.1"
```

--------------------------------

### Year Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a year interval literal.

```sql
INTERVAL -'1999' YEAR
```

--------------------------------

### Year to Month Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for a year to month interval literal.

```sql
INTERVAL '-1999-11' YEAR TO MONTH
```

--------------------------------

### Vortex Filter Pushdown Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/vortex/index.md

Illustrates filter pushdown where a comparison filter is pushed to Vortex, and an `isNotNull` filter is applied post-read.

```python
df = spark.read.format("vortex").option("path", "/data/events.vortex").load()

# The comparison filter is pushed down to Vortex.
# The `isNotNull` filter is applied post-read.
df.filter((df.score > 90.0) & (df.name.isNotNull())).show()
```

--------------------------------

### Install Specific Protobuf and gRPC Versions for PySpark Connect

Source: https://github.com/lakehq/sail/blob/main/docs/guide/troubleshooting/index.md

Install specific versions of Protobuf and gRPC libraries alongside PySpark to resolve Protobuf version mismatch warnings when using PySpark 4.0 in Spark Connect mode.

```bash
pip install "pyspark[connect]==4.0.0" \
  "protobuf==5.28.3" \
  "grpcio==1.71.2" \
  "grpcio-status==1.71.2"
```

--------------------------------

### Preview Documentation Site Locally

Source: https://github.com/lakehq/sail/blob/main/docs/development/documentation/index.md

Builds and serves the documentation site locally using VitePress. This allows you to preview the site before deployment.

```bash
pnpm run docs:preview
```

--------------------------------

### Start Local PySpark Shell

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/pyspark-local.md

Use this command to run the PySpark shell with the default Java implementation for local development.

```bash
hatch run pyspark
```

--------------------------------

### SQL DATE Literal Examples

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/datetime.md

Demonstrates the use of DATE literals with different levels of specificity for year, month, and day.

```sql
SELECT DATE '2001' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2001-01-01|
-- +----------+

SELECT DATE '2005-07' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2005-07-01|
-- +----------+

SELECT DATE '2019-12-25' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2019-12-25|
-- +----------+
```

--------------------------------

### Hour to Second Interval Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Demonstrates the syntax for an hour to second interval literal, including fractional seconds.

```sql
INTERVAL '123:10:59' HOUR TO SECOND
```

--------------------------------

### Run Spark Connect Server in Specific Environment

Source: https://github.com/lakehq/sail/blob/main/docs/development/spark-tests/test-spark.md

Starts the Sail Spark Connect server within a specific test-spark environment to ensure consistency between the server and tests. This may increase build and start time due to cache pollution.

```bash
hatch run test-spark.spark-3.5.7:scripts/spark-tests/run-server.sh
```

--------------------------------

### Configure Unity Catalog Provider via Environment Variables

Source: https://github.com/lakehq/sail/blob/main/docs/guide/catalog/unity.md

This example shows how to set environment variables to configure the Unity Catalog provider. It specifies that HTTP URLs are not allowed and defines the catalog provider configuration, including its type, name, URI, default catalog, and authentication token.

```bash
export UNITY_ALLOW_HTTP_URL='false'
export SAIL_CATALOG__LIST='[{type="unity", name="sail", uri="https://catalog.example.com", default_catalog="meow", token="..."}]'
```

--------------------------------

### Multi-unit Interval of Weeks

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Example of creating an interval using weeks as the unit.

```sql
SELECT INTERVAL 3 WEEK AS col;
-- +-----------------+
-- |col              |
-- +-----------------+
-- |INTERVAL '21' DAY|
-- +-----------------+

```

--------------------------------

### Vortex SQL Query Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/vortex/index.md

Shows how to query Vortex data using SQL by creating a temporary view from the DataFrame.

```python
df = spark.read.format("vortex").option("path", "/data/events.vortex").load()
df.createOrReplaceTempView("events")

spark.sql("SELECT * FROM events WHERE id > 100 ORDER BY score DESC").show()
```

--------------------------------

### Run Spark Connect Server

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/benchmark.md

Starts the Spark Connect server in release mode with logging enabled. This command is used to establish a connection for PySpark applications.

```shell
env RUST_LOG=info BENCHMARK=1 scripts/spark-tests/run-server.sh
```

--------------------------------

### Run Sail Spark Connect Server

Source: https://github.com/lakehq/sail/blob/main/docs/introduction/getting-started/index.md

Start the Sail Spark Connect server with informational logging enabled. The server defaults to listening on localhost:50051.

```bash
env RUST_LOG=info sail spark server
```

--------------------------------

### Negative DECIMAL Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a negative DECIMAL literal. The type is inferred as DECIMAL.

```sql
SELECT -0.4321 AS col;
-- +--------+
-- |     col|
-- +--------+
-- |-0.4321 |
-- +--------+
```

--------------------------------

### SMALLINT Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a SMALLINT literal using the 'S' postfix. This represents a 2-byte signed integer.

```sql
SELECT 512S AS col;
-- +----+
-- |col |
-- +----+
-- |512 |
-- +----+
```

--------------------------------

### Run Spark Connect Server with Object Store Credentials

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/object-store.md

Command to launch the Spark Connect server, setting environment variables for AWS S3, Azure Storage, and Google Cloud Storage compatibility with local emulators or public buckets.

```bash
env \
  AWS_ACCESS_KEY_ID="sail" \
  AWS_SECRET_ACCESS_KEY="password" \
  AWS_ENDPOINT="http://localhost:19000" \
  AWS_VIRTUAL_HOSTED_STYLE_REQUEST="false" \
  AWS_ALLOW_HTTP="true" \
  AZURE_STORAGE_ACCOUNT_NAME="devstoreaccount1" \
  AZURE_STORAGE_ACCOUNT_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" \
  AZURE_STORAGE_ENDPOINT="http://localhost:10000/devstoreaccount1" \
  AZURE_STORAGE_USE_EMULATOR="true" \
  GOOGLE_SKIP_SIGNATURE="true" \
  HADOOP_USER_NAME="sail" \
  hatch run scripts/spark-tests/run-server.sh
```

--------------------------------

### Run Tests on Installed Package

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/python.md

Runs pytest specifically on the installed package, useful for testing the distributed version.

```bash
hatch run pytest --pyargs pysail
```

--------------------------------

### BIGINT Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a BIGINT literal using the 'L' postfix. This represents an 8-byte signed integer.

```sql
SELECT 9223372036854775807L AS col;
-- +-------------------+
-- |                col|
-- +-------------------+
-- |9223372036854775807|
-- +-------------------+
```

--------------------------------

### Start PySpark Shell with Spark Connect

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/pyspark-local.md

Run the PySpark shell with the Spark Connect implementation enabled, specifying the connection mode and remote address.

```bash
env SPARK_CONNECT_MODE_ENABLED=1 SPARK_REMOTE="sc://localhost:50051" hatch run pyspark
```

--------------------------------

### SQL TIMESTAMP Literal Examples

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/datetime.md

Illustrates TIMESTAMP literals with various components including date, time, fractional seconds, and time zones.

```sql
SELECT TIMESTAMP '2000-02-29 23:59:59.123' AS col;
-- +-----------------------+
-- |                    col|
-- +-----------------------+
-- |2000-02-29 23:59:59.123|
-- +-----------------------+

SELECT TIMESTAMP '2015-06-30 12:00:00.999999UTC-05:00' AS col;
-- +--------------------------+
-- |                      col |
-- +--------------------------+
-- |2015-06-30 17:00:00.999999|
-- +--------------------------+

SELECT TIMESTAMP '2010-08' AS col;
-- +-------------------+
-- |                col|
-- +-------------------+
-- |2010-08-01 00:00:00|
-- +-------------------+
```

--------------------------------

### Launch MCP Server (Standard Method)

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/mcp-server.md

Launches the MCP server using the installed Sail CLI. This method requires the Sail binary to be rebuilt if changes are made to the MCP server Python script.

```bash
sail spark mcp-server
```

--------------------------------

### Launch MCP Inspector

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/mcp-server.md

Installs and launches the MCP Inspector tool, which is used for debugging and interacting with the MCP server. The tool will output its local URL upon startup.

```bash
pnpx @modelcontextprotocol/inspector
```

--------------------------------

### Create a Local Kubernetes Cluster with kind

Source: https://github.com/lakehq/sail/blob/main/docs/guide/deployment/kubernetes.md

Creates a local Kubernetes cluster using the kind tool.

```bash
kind create cluster
```

--------------------------------

### List All Catalogs and Inspect

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/catalog/test_list_catalogs.txt

Lists all available catalogs and checks their properties. Assumes SparkSession is already initialized.

```python
catalogs = {c.name: c for c in spark.catalog.listCatalogs()}
len(catalogs) == 2
"system" in catalogs and catalogs["system"].description is None
user_catalogs = catalogs.keys() - {"system"}
len(user_catalogs) == 1 and next(iter(user_catalogs)) in {"sail", "sail_catalog", "spark_catalog"}
all(catalogs[name].description is None for name in user_catalogs)
```

--------------------------------

### Run Arrow Flight SQL Server

Source: https://github.com/lakehq/sail/blob/main/docs/guide/cli/index.md

Start Sail as an Arrow Flight SQL server, configuring the IP address and port.

```bash
sail flight server --ip 127.0.0.1 --port 32010
```

--------------------------------

### DOUBLE Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a DOUBLE literal using the 'D' postfix. This represents an 8-byte double-precision floating point number.

```sql
SELECT 6.9D AS col;
-- +-----+
-- | col |
-- +-----+
-- | 6.9 |
-- +-----+
```

--------------------------------

### SQL Binary Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/binary.md

Demonstrates the usage of a binary literal in a SQL SELECT statement. The literal is specified using the X'...' syntax with hexadecimal digits.

```sql
SELECT X'123456' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |[12 34 56]|
-- +----------+
```

--------------------------------

### Develop Python Package for Documentation

Source: https://github.com/lakehq/sail/blob/main/docs/development/documentation/index.md

Installs the Python package in editable mode within the 'docs' Hatch environment. This command is necessary after making changes to the Rust code.

```bash
hatch run docs:maturin develop
```

--------------------------------

### Partitioned Delta Table Operations (SQL)

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/delta/examples.md

Demonstrates creating, inserting into, and selecting from a partitioned Delta table using SQL. Partitioning organizes data into directories based on column values.

```sql
CREATE TABLE metrics (year INT, value FLOAT)
USING delta
LOCATION 'file:///tmp/sail/metrics'
PARTITIONED BY (year);

INSERT INTO metrics VALUES (2024, 1.0), (2025, 2.0);

SELECT * FROM metrics WHERE year > 2024;
```

--------------------------------

### PySpark Skill Definition

Source: https://github.com/lakehq/sail/blob/main/docs/guide/integrations/agent-skills.md

This is an example of a skill definition file (SKILL.md) that exposes the `sail spark run` command. Refer to your LLM provider's documentation for instructions on loading this skill.

```markdown
name: pyspark

description: Run PySpark code on Sail's compute engine.

parameters:
  - name: code
    description: The PySpark code to run.
    required: true
    type: string

run: |
  sail spark run --code "{{code}}"
```

--------------------------------

### DOUBLE Literal with Exponent Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a DOUBLE literal using scientific notation with an exponent. The 'D' postfix is optional when an exponent is present.

```sql
SELECT 8.21E1 AS col, TYPEOF(8.21E1) AS type;
-- +------+------+
-- |   col|  type|
-- +------+------+
-- | 82.1 |double|
-- +------+------+
```

--------------------------------

### Build Sail Python Package with CPU Optimization

Source: https://github.com/lakehq/sail/blob/main/docs/introduction/installation/index.md

Install Sail from source with CPU-specific optimizations by setting the RUSTFLAGS environment variable. This command also enables verbose output and prevents the use of pre-built wheels.

```bash-vue
env RUSTFLAGS="-C target-cpu=native" pip install "pysail=={{ libVersion }}" -v --no-binary pysail
```

--------------------------------

### Install Patched PySpark Package

Source: https://github.com/lakehq/sail/blob/main/docs/development/spark-tests/test-spark.md

Installs the patched PySpark package for specific Spark versions into the test-spark matrix environments. The environment is created automatically if it doesn't exist.

```bash
hatch run test-spark.spark-4.1.1:install-pyspark
hatch run test-spark.spark-3.5.7:install-pyspark
```

--------------------------------

### Start Sail Flight SQL Server Programmatically (Python)

Source: https://github.com/lakehq/sail/blob/main/docs/guide/integrations/flight-sql.md

Start the Arrow Flight SQL server programmatically using the Sail Python API. This allows for integration into Python applications.

```python
from pysail.flight import FlightSqlServer

server = FlightSqlServer(ip="127.0.0.1", port=32010)
server.start(background=False)
```

--------------------------------

### Prioritize Rustup-Managed Rust Installation

Source: https://github.com/lakehq/sail/blob/main/docs/development/setup/prerequisites.md

Add the rustup-managed Cargo bin directory to the PATH environment variable to ensure it takes precedence over other Rust installations, by adding the export command to your shell profile.

```bash
export PATH="$HOME/.cargo/bin:$PATH"
```

--------------------------------

### Build VitePress Documentation Site

Source: https://github.com/lakehq/sail/blob/main/docs/development/documentation/index.md

Builds the static documentation site using VitePress. This command generates the final HTML, CSS, and JavaScript files for deployment.

```bash
pnpm run docs:build
```

--------------------------------

### Apply Customized Kubernetes Manifests with Kustomize

Source: https://github.com/lakehq/sail/blob/main/docs/guide/deployment/kubernetes.md

Applies customized Kubernetes manifests using Kustomize from the k8s/ directory.

```bash
kubectl apply -k k8s/
```

--------------------------------

### Create and List Temporary View

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/catalog/test_temp_view.txt

Creates a temporary view named 't1' and then lists all tables in the catalog to verify its creation.

```python
>>> spark.range(1).createTempView("t1")
>>> spark.catalog.listTables()
[Table(name='t1', catalog=None, namespace=[], description=None, tableType='TEMPORARY', isTemporary=True)]
```

--------------------------------

### Date Subtraction

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/literal/test_date.txt

Calculates the interval between two dates by subtracting them. Note: This example is skipped in doctests.

```python
spark.sql("SELECT DATE '1970-01-01' - DATE '-1970-01-01'").show(truncate=False)
```

--------------------------------

### Configure AWS Glue Catalog with Caching

Source: https://github.com/lakehq/sail/blob/main/docs/guide/catalog/glue.md

Enable caching for database and table listings by setting cache types, sizes, and TTL values. Caching improves performance by reducing repeated requests.

```bash
export SAIL_CATALOG__LIST='[{type="glue", name="sail", region="us-west-2", database_cache_type="global", database_cache_size=100, database_cache_ttl_secs=3600, table_cache_type="global", table_cache_size=1000, table_cache_ttl_secs=3600}]'
```

--------------------------------

### SQL Interval Literal: Second

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Example of an interval literal specifying seconds with fractional parts.

```sql
INTERVAL '2000.000002' SECOND
```

--------------------------------

### SQL Interval Literal: Minute to Second

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Example of an interval literal specifying minutes and seconds.

```sql
INTERVAL '2000:02.002' MINUTE TO SECOND
```

--------------------------------

### Build and Run Sail Spark Connect Server

Source: https://github.com/lakehq/sail/blob/main/docs/development/spark-tests/test-ibis.md

Use this command to build and run the Sail Spark Connect server in the 'test-ibis' environment. This is a prerequisite for running Ibis tests.

```bash
hatch run test-ibis:scripts/spark-tests/run-server.sh
```

--------------------------------

### Basic Iceberg Write and Read (SQL)

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/iceberg/examples.md

Demonstrates how to create, insert data into, and select from an Iceberg table using Spark SQL.

```sql
CREATE TABLE users (id INT, name STRING)
USING iceberg
LOCATION 'file:///tmp/sail/users';

INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob');

SELECT * FROM users;
```

--------------------------------

### Configure Memory and Iceberg-REST Catalogs

Source: https://github.com/lakehq/sail/blob/main/docs/guide/catalog/index.md

Set environment variables to configure a list of catalog providers, including a memory catalog and an Iceberg REST catalog, and specify the default catalog.

```bash
export SAIL_CATALOG__LIST='[{name="c1", type="memory", initial_database=["default"]}, {name="c2", type="iceberg-rest", uri="https://catalog.example.com"}]'
export SAIL_CATALOG__DEFAULT_CATALOG="c1"
```

--------------------------------

### Integer Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a negative integer literal. The default type is INT.

```sql
SELECT -2147483600 AS col;
-- +-----------+
-- |        col|
-- +-----------+
-- |-2147483600|
-- +-----------+
```

--------------------------------

### Run Sail Spark Server

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/standalone-binary.md

Execute the standalone Sail Spark Connect server binary. The `--help` option can be used to discover all supported arguments.

```bash
target/release/sail spark server
```

--------------------------------

### TINYINT Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a TINYINT literal using the 'Y' postfix. This represents a 1-byte signed integer.

```sql
SELECT -64Y AS col;
-- +----+
-- | col|
-- +----+
-- |-64 |
-- +----+
```

--------------------------------

### Create a kind Cluster with Custom Configuration

Source: https://github.com/lakehq/sail/blob/main/docs/guide/deployment/kubernetes.md

Creates a local Kubernetes cluster using kind with a specified configuration file.

```bash
kind create cluster --config k8s/kind-config.yaml
```

--------------------------------

### Example of Delimited Identifier Usage

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/identifiers.md

Illustrates the use of delimited identifiers to query a table with a name containing a dot.

```sql
SELECT * FROM `a.b`
```

--------------------------------

### SQL NULL Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/null.md

Demonstrates how to select a NULL value as a column in SQL. The NULL literal is case-insensitive.

```sql
SELECT NULL AS col;
-- +----+--
-- | col|
-- +----+--
-- |NULL|
-- +----+
```

--------------------------------

### Create and Show Temporary View 'alltypes'

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/aggregate/test_agg_max_by_min_by.txt

Creates a temporary view named 'alltypes' with integer and bigint columns derived from a sequence.

```python
spark.sql("""
CREATE OR REPLACE TEMP VIEW alltypes AS
SELECT
  CAST(v AS INT) AS int_col,
  CAST(v % 3 AS BIGINT) AS bigint_col
FROM (SELECT explode(sequence(0, 9)) AS v)
""")
```

--------------------------------

### Spark SQL COUNT with GROUP BY

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/test_group_by_alias.txt

Calculates the count of column 'b' and groups the results by column 'a'. This is a basic aggregation example.

```python
spark.sql("SELECT count(b) AS a FROM VALUES (1, 1), (1, 2) AS t(a, b) GROUP BY a").show()
```

--------------------------------

### Basic Delta Table Operations (SQL)

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/delta/examples.md

Demonstrates creating, inserting into, and selecting from a Delta table using SQL. Ensure a Spark session is available.

```sql
CREATE TABLE users (id INT, name STRING)
USING delta
LOCATION 'file:///tmp/sail/users';

INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob');

SELECT * FROM users;
```

--------------------------------

### Configure LLVM Linker on Windows

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/rust.md

Create a Cargo configuration file to use the LLVM linker for Rust projects on Windows when using the MSVC target environment.

```toml
[target.'cfg(all(windows, target_env = "msvc"))']
linker = "lld-link"
```

--------------------------------

### DECIMAL Literal Example

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/numeric.md

Demonstrates selecting a DECIMAL literal with a decimal point. The type is inferred as DECIMAL(precision, scale).

```sql
SELECT 76.543 AS col, TYPEOF(76.543) AS type;
-- +-------+------------+
-- |    col|        type|
-- +-------+------------+
-- | 76.543|decimal(5,3)|
-- +-------+------------+
```

--------------------------------

### Partitioned Iceberg Write and Read (SQL)

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/iceberg/examples.md

Demonstrates creating, inserting into, and querying partitioned Iceberg tables using Spark SQL. Tables are partitioned by the 'year' column.

```sql
CREATE TABLE metrics (year INT, value FLOAT)
USING iceberg
LOCATION 'file:///tmp/sail/metrics'
PARTITIONED BY (year);

INSERT INTO metrics VALUES (2024, 1.0), (2025, 2.0);

SELECT * FROM metrics WHERE year > 2024;
```

--------------------------------

### Basic Vortex Read and Show

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/vortex/index.md

Demonstrates a basic read operation for a Vortex file and displays the resulting DataFrame.

```python
from pysail.spark.datasource.vortex import VortexDataSource

spark.dataSource.register(VortexDataSource)

df = spark.read.format("vortex").option("path", "/data/events.vortex").load()
df.show()
```

--------------------------------

### Run Tests Against Specific Spark Version

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/python.md

Executes pytest against a specific Spark version (e.g., 3.5.7) after installing the package in that environment.

```bash
hatch run test.spark-3.5.7:pytest --pyargs pysail
```

--------------------------------

### Basic Delta Table Operations (Python)

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/delta/examples.md

Demonstrates creating, overwriting, appending, and reading a Delta table using Python with Spark. Ensure a Spark session is available.

```python
path = "file:///tmp/sail/users"
df = spark.createDataFrame(
    [(1, "Alice"), (2, "Bob")],
    schema="id INT, name STRING",
)

# This creates a new table or overwrites an existing one.
df.write.format("delta").mode("overwrite").save(path)
# This appends data to an existing table.
df.write.format("delta").mode("append").save(path)

df = spark.read.format("delta").load(path)
df.show()
```

--------------------------------

### Interact with Catalogs using SQL

Source: https://github.com/lakehq/sail/blob/main/docs/guide/catalog/index.md

Execute SQL statements to display the current catalog and list tables within it.

```sql
-- show the current catalog
SELECT current_catalog()
-- show tables in the current catalog
SHOW TABLES
```

--------------------------------

### Division of Interval by Integer

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/function/test_math_try_div.txt

Shows how to use try_divide with PySpark's make_interval function. This example divides a 1-day interval by 2.

```python
>>> spark.range(1).select(sf.try_divide(sf.make_interval(days=sf.lit(1)), sf.lit(2))).show()
+-------------------------------------------------+
|try_divide(make_interval(0, 0, 0, 1, 0, 0, 0), 2)|
+-------------------------------------------------+
|                                         12 hours|
+-------------------------------------------------+
```

--------------------------------

### Fill NaN in FloatType Column

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/function/test_fillna.txt

Replaces NaN values in a FloatType column with 0. This is similar to the DoubleType example but uses FloatType.

```python
from pyspark.sql import types as T

schema = T.StructType([T.StructField("value", T.FloatType(), True)])
df = spark.createDataFrame([(float("nan"),), (1.5,)], schema)
df.fillna(0).collect()
```

--------------------------------

### Get Column Names with Special Characters

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/test_ipython_key_completions.txt

Demonstrates that _ipython_key_completions_ correctly handles column names containing spaces or special characters.

```python
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], ["age 1", "name?1"])
>>> df._ipython_key_completions_()
['age 1', 'name?1']
```

--------------------------------

### Define and Use Fibonacci UDTF in PySpark

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/udf/test_udtf.txt

Defines a Fibonacci UDTF and applies it to a literal value in PySpark. Ensure PySpark is installed and configured.

```python
from pyspark.sql.functions import lit, udtf

class Fibonacci:
    def eval(self, n: int):
        a, b = 0, 1
        for _ in range(n):
            a, b = b, a + b
            yield (a,)

fibonacci = udtf(Fibonacci, returnType="value: integer")
fibonacci(lit(5)).show()
```

--------------------------------

### Create Git Tag

Source: https://github.com/lakehq/sail/blob/main/docs/development/release/index.md

Create a Git tag locally and push it to the remote repository to initiate the release process.

```bash
git tag $TAG
git push origin tag $TAG
```

--------------------------------

### Build and Test Rust Code

Source: https://github.com/lakehq/sail/blob/main/docs/development/build/rust.md

Run this command to format, lint, build, and test the Rust code. It includes a nightly toolchain for formatting and an environment variable to update test gold data.

```bash
cargo +nightly fmt && \
  cargo clippy --all-targets --all-features && \
  cargo build && \
  env SAIL_UPDATE_GOLD_DATA=1 cargo test
```

--------------------------------

### Get Column Names from DataFrame

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/test_ipython_key_completions.txt

Use _ipython_key_completions_ to list column names of a Spark DataFrame. This is useful for interactive exploration in IPython/Jupyter environments.

```python
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], ["age", "name"])
>>> df._ipython_key_completions_()
['age', 'name']
```

--------------------------------

### Creating Intervals with Fractional Seconds

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/function/test_datetime_make_dt_interval.txt

Shows how to create a DAY TO SECOND interval that includes fractional seconds. This is useful for precise time interval representations.

```python
spark.sql("SELECT (make_dt_interval(0, 0, 0, 0.1))").show(truncate=False)
```

--------------------------------

### Start PySpark Shell with SPARK_LOCAL_IP

Source: https://github.com/lakehq/sail/blob/main/docs/development/recipes/pyspark-local.md

If you encounter a Spark driver address binding error, set the SPARK_LOCAL_IP environment variable to explicitly bind to localhost.

```bash
env SPARK_LOCAL_IP=127.0.0.1 hatch run pyspark
```

--------------------------------

### List Catalogs with No Matches

Source: https://github.com/lakehq/sail/blob/main/python/pysail/tests/spark/catalog/test_list_catalogs.txt

Demonstrates listing catalogs with a wildcard pattern that yields no results. Assumes SparkSession is available.

```python
spark.catalog.listCatalogs("hive*")
```

--------------------------------

### Partitioned Delta Table Operations (Python)

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sources/delta/examples.md

Shows how to write and read partitioned Delta tables using Python with Spark. Partitioning improves query performance by organizing data into directories.

```python
path = "file:///tmp/sail/metrics"
df = spark.createDataFrame(
    [(2024, 1.0), (2025, 2.0)],
    schema="year INT, value FLOAT",
)

df.write.format("delta").mode("overwrite").partitionBy("year").save(path)

df = spark.read.format("delta").load(path).filter("year > 2024")
df.show()
```

--------------------------------

### Complex Multi-unit Interval

Source: https://github.com/lakehq/sail/blob/main/docs/guide/sql/literals/interval.md

Shows a comprehensive example combining multiple units including weeks, days, hours, minutes, seconds, milliseconds, and microseconds.

```sql
SELECT INTERVAL 3 WEEK 4 DAYS 5 HOUR 6 MINUTES 7 SECOND 8 MILLISECOND 9 MICROSECONDS AS col;
-- +-------------------------------------------+
-- |col                                        |
-- +-------------------------------------------+
-- |INTERVAL '25 05:06:07.008009' DAY TO SECOND|
-- +-------------------------------------------+

```

--------------------------------

### Basic Iceberg REST Catalog Configuration

Source: https://github.com/lakehq/sail/blob/main/docs/guide/catalog/iceberg-rest.md

Configure a basic Iceberg REST catalog by specifying its type, name, and URI. This is the minimum required configuration.

```bash
export SAIL_CATALOG__LIST='[{type="iceberg-rest", name="sail", uri="https://catalog.example.com"}]'
```