### Using the Sail Library to Start a Spark Connect Server and Connect

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Example Python script demonstrating how to start a Sail Spark Connect server and connect to it using PySpark.

```python
from pysail.spark import SparkConnectServer
from pyspark.sql import SparkSession

server = SparkConnectServer()
server.start()
_, port = server.listening_address

spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate()

# Run a simple query
spark.sql("SELECT 1 + 1").show()

# Use the DataFrame API to read multimodal data
spark.read.format("binaryFile").option("pathGlobFilter", "*.png").load("/path/to/data").show()

# Use Spark SQL to create a table that refers to multimodal data in an object storage
spark.sql("""
CREATE TABLE pdfs
USING binaryFile
OPTIONS (pathGlobFilter '*.pdf')
LOCATION 's3://my-bucket/path/to/data'
""")
spark.sql("SELECT * FROM pdfs").show()
```

--------------------------------

### Install Sail from PyPI

Source: https://docs.lakesail.com/sail/latest/introduction/installation

Quickly get started by installing Sail from PyPI using pip.

```bash
pip install "pysail==0.6.3"
```

--------------------------------

### Running the Sail Spark Connect Server

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Command to start a Spark Connect server powered by Sail with info logging enabled.

```bash
env RUST_LOG=info sail spark server
```

--------------------------------

### Kind Cluster Configuration Example

Source: https://docs.lakesail.com/sail/latest/guide/deployment/kubernetes.html

Example configuration file for creating a kind cluster with custom mounts.

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /private/tmp/sail
        containerPath: /private/tmp/sail
      - hostPath: /tmp/sail
        containerPath: /tmp/sail
```

--------------------------------

### Install Dependencies

Source: https://docs.lakesail.com/sail/latest/development/documentation

Installs the necessary dependencies for building the documentation using pnpm.

```bash
pnpm install
```

--------------------------------

### Run the Spark Connect server

Source: https://docs.lakesail.com/sail/latest/development/recipes/standalone-binary.html

Starts the Spark Connect server using the built standalone binary.

```bash
target/release/sail spark server
```

--------------------------------

### Package Installation for Spark 3.5

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Installs the pysail package and pyspark with connect dependencies for Spark 3.5.

```bash
pip install "pysail==0.6.3"
pip install "pyspark[connect]==3.5.7
```

--------------------------------

### Package Installation for Spark 4.1 (Client)

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Installs the pysail package and the pyspark-client for Spark 4.1.

```bash
pip install "pysail==0.6.3"
pip install "pyspark-client==4.1.1"
```

--------------------------------

### Install LLVM on Windows

Source: https://docs.lakesail.com/sail/latest/development/build/rust.html

Command to install LLVM using scoop on Windows.

```bash
scoop install llvm
```

--------------------------------

### Start VitePress Development Server

Source: https://docs.lakesail.com/sail/latest/development/documentation

Starts the development server for VitePress, with automatic reflection of Markdown file changes.

```bash
pnpm run docs:dev
```

--------------------------------

### FlightSqlServer start method

Source: https://docs.lakesail.com/sail/latest/reference/python/api/pysail.flight.FlightSqlServer.html

Start the server.

```python
start(_*_ , _background =True_)
```

--------------------------------

### Multi-unit Interval Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/interval.html

Examples demonstrating the multi-unit interval syntax with various units and combinations.

```sql
SELECT INTERVAL 3 WEEK AS col;
-- +-----------------+
-- |col              |
-- +-----------------+
-- |INTERVAL '21' DAY|
-- +-----------------+

SELECT INTERVAL -2 WEEKS '3' DAYS AS col;
-- +------------------+
-- |col               |
-- +------------------+
-- |INTERVAL '-11' DAY|
-- +------------------+

SELECT INTERVAL '3 DAYS 50 SECONDS' AS col;
-- +-----------------------------------+
-- |col                                |
-- +-----------------------------------+
-- |INTERVAL '3 00:00:50' DAY TO SECOND|
-- +-----------------------------------+

SELECT INTERVAL 3 WEEK 4 DAYS 5 HOUR 6 MINUTES 7 SECOND 8 MILLISECOND 9 MICROSECONDS AS col;
-- +-------------------------------------------+
-- |col                                        |
-- +-------------------------------------------+
-- |INTERVAL '25 05:06:07.008009' DAY TO SECOND|
-- +-------------------------------------------+
```

--------------------------------

### Package Installation for Spark 4.1 (using pyspark[connect])

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Installs the pysail package and pyspark with connect dependencies for Spark 4.1.

```bash
pip install "pysail==0.6.3"
pip install "pyspark[connect]==4.1.1"
```

--------------------------------

### Kustomize Example for Sail Workers

Source: https://docs.lakesail.com/sail/latest/reference/changelog

An example demonstrating the use of Kustomize with pod templates for Sail workers.

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

patchesStrategicMerge:
  - sail-worker-patch.yaml

```

--------------------------------

### Start Docker Compose Services

Source: https://docs.lakesail.com/sail/latest/development/recipes/object-store.html

Starts all the services defined in the compose.yml file in the background.

```bash
docker compose up -d
```

--------------------------------

### Install Rust Toolchains via rustup

Source: https://docs.lakesail.com/sail/latest/development/setup/prerequisites.html

Commands to install the stable and nightly Rust toolchains, including the `llvm-tools-preview` component for the stable toolchain.

```bash
rustup toolchain install stable --profile default --component llvm-tools-preview
rustup toolchain install nightly --profile default
```

--------------------------------

### Install Rustup and Node.js on macOS via Homebrew

Source: https://docs.lakesail.com/sail/latest/development/setup/prerequisites.html

Commands to install Rustup and Node.js using Homebrew on macOS.

```bash
brew install rustup
brew install node
```

--------------------------------

### ANSI Interval Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/interval.html

Examples demonstrating the ANSI interval syntax for year-month and day-time intervals.

```sql
SELECT INTERVAL '5-6' YEAR TO MONTH AS col;
-- +----------------------------+
-- |col                         |
-- +----------------------------+
-- |INTERVAL '5-6' YEAR TO MONTH|
-- +----------------------------+

SELECT INTERVAL -'12 23:45:59.888888' DAY TO SECOND AS col;
-- +--------------------------------------------+
-- |col                                         |
-- +--------------------------------------------+
-- |INTERVAL '-12 23:45:59.888888' DAY TO SECOND|
-- +--------------------------------------------+
```

--------------------------------

### Install Development Tools on macOS via Homebrew

Source: https://docs.lakesail.com/sail/latest/development/setup/prerequisites.html

Commands to install Protocol Buffers compiler, Hatch, Maturin, Zig, and pnpm using Homebrew on macOS.

```bash
brew install protobuf hatch maturin zig pnpm
```

--------------------------------

### Quick Examples

Source: https://docs.lakesail.com/sail/latest/guide/storage

Examples of reading and writing data from various storage systems using Spark and Sail.

```python
# Local file system
df = spark.read.parquet("/path/to/local/data")
df = spark.read.parquet("file:///path/to/local/data")

# Cloud storage
df = spark.read.parquet("s3://bucket/data")
df = spark.read.parquet("azure://container/data")
df = spark.read.parquet("gs://bucket/data")

# In-memory storage
df = spark.read.parquet("memory:///cached/data")

# HTTP/HTTPS endpoints
df = spark.read.json("https://api.example.com/data.json")

# Create tables from any storage
spark.sql("""
CREATE TABLE my_table (id INT, name STRING)
USING parquet
LOCATION 's3://bucket/path/to/data'
""")
```

--------------------------------

### Example Skill Definition

Source: https://docs.lakesail.com/sail/latest/guide/integrations/agent-skills.html

This is a basic example of a skill definition for running PySpark scripts.

```yaml
---
name: run-pyspark-script
description: Runs a PySpark script.
---
```

--------------------------------

### Installing the Built Python Package

Source: https://docs.lakesail.com/sail/latest/development/build/python.html

Installs the previously built Python package into the default environment.

```bash
hatch run install-pysail
```

--------------------------------

### Install Azure Storage Client Libraries

Source: https://docs.lakesail.com/sail/latest/development/recipes/object-store.html

Installs the necessary Python client libraries for Azure Blob Storage and DataLake Storage.

```bash
pip install azure-storage-blob azure-storage-file-datalake
```

--------------------------------

### Installing the Package in All Test Environments

Source: https://docs.lakesail.com/sail/latest/development/build/python.html

Installs the package in all `test` matrix environments for running tests against multiple Spark versions.

```bash
hatch run test:install-pysail
```

--------------------------------

### Install OpenAPI Generator CLI

Source: https://docs.lakesail.com/sail/latest/development/recipes/iceberg-openapi-generation.html

Install the OpenAPI Generator CLI using Homebrew.

```bash
brew install openapi-generator
```

--------------------------------

### Installing the Package in a Specific Test Environment

Source: https://docs.lakesail.com/sail/latest/development/build/python.html

Installs the package into a specific `test` environment for testing against a particular Spark version.

```bash
hatch run test.spark-3.5.7:install-pysail
```

--------------------------------

### Prioritize rustup-managed Rust installation

Source: https://docs.lakesail.com/sail/latest/development/setup/prerequisites.html

Shell command to add to your shell profile to prioritize the rustup-managed Rust installation if Homebrew overrides it.

```bash
export PATH="$HOME/.cargo/bin:$PATH"
```

--------------------------------

### Spark SQL Example

Source: https://docs.lakesail.com/sail/latest/guide/sql

This example demonstrates how to create a temporary view from a DataFrame and run SQL queries against it using a Spark client session connected to the Sail server.

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:50051").getOrCreate()

df = spark.read.parquet("/data/users.parquet")
df.createOrReplaceTempView("users")

spark.sql("SELECT name, email FROM users WHERE age > 30").show()
```

--------------------------------

### Run PySpark shell with Java implementation

Source: https://docs.lakesail.com/sail/latest/development/recipes/pyspark-local.html

This command starts the PySpark shell using the Java implementation.

```bash
hatch run pyspark
```

--------------------------------

### Integral Number Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/numeric.html

Examples demonstrating the syntax and behavior of integral numeric literals with different postfixes (L, S, Y) and without any postfix.

```sql
SELECT -2147483600 AS col;
-- +-----------+
-- |        col|
-- +-----------+
-- |-2147483600|
-- +-----------+

SELECT 9223372036854775807L AS col;
-- +-------------------+
-- |                col|
-- +-------------------+
-- |9223372036854775807|
-- +-------------------+

SELECT -64Y AS col;
-- +----+
-- | col|
-- +----+
-- |-64 |
-- +----+

SELECT 512S AS col;
-- +----+
-- |col |
-- +----+
-- |512 |
-- +----+
```

--------------------------------

### Preview Documentation Site Locally

Source: https://docs.lakesail.com/sail/latest/development/documentation

Starts a local server to preview the documentation site.

```bash
pnpm run docs:preview
```

--------------------------------

### Data Partitioning

Source: https://docs.lakesail.com/sail/latest/guide/sources/delta/examples.html

Shows how to work with partitioned Delta tables using Spark DataFrame API and SQL, improving query performance.

```python
path = "file:///tmp/sail/metrics"
df = spark.createDataFrame(
    [(2024, 1.0), (2025, 2.0)],
    schema="year INT, value FLOAT",
)

df.write.format("delta").mode("overwrite").partitionBy("year").save(path)

df = spark.read.format("delta").load(path).filter("year > 2024")
df.show()
```

```sql
CREATE TABLE metrics (year INT, value FLOAT)
USING delta
LOCATION 'file:///tmp/sail/metrics'
PARTITIONED BY (year);

INSERT INTO metrics VALUES (2024, 1.0), (2025, 2.0);

SELECT * FROM metrics WHERE year > 2024;
```

--------------------------------

### Build Docker Image

Source: https://docs.lakesail.com/sail/latest/guide/deployment/docker-images/quick-start.html

Command to build the Docker image for Sail.

```bash
docker build -t sail:latest --build-arg PYSAIL_VERSION=0.6.3 .
```

--------------------------------

### Basic Usage

Source: https://docs.lakesail.com/sail/latest/guide/sources/delta/examples.html

Demonstrates creating, writing, and reading a Delta table using both Python and SQL.

```python
path = "file:///tmp/sail/users"
df = spark.createDataFrame(
    [(1, "Alice"), (2, "Bob")],
    schema="id INT, name STRING",
)

# This creates a new table or overwrites an existing one.
df.write.format("delta").mode("overwrite").save(path)
# This appends data to an existing table.
df.write.format("delta").mode("append").save(path)

df = spark.read.format("delta").load(path)
df.show()
```

```sql
CREATE TABLE users (id INT, name STRING)
USING delta
LOCATION 'file:///tmp/sail/users';

INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob');

SELECT * FROM users;
```

--------------------------------

### Install Multiple Python Versions with pyenv

Source: https://docs.lakesail.com/sail/latest/development/setup/python.html

This command installs specific patch versions of Python 3.8, 3.9, 3.10, and 3.11 using pyenv, making them available for Hatch environments.

```bash
pyenv local 3.8.19 3.9.19 3.10.14 3.11.9
```

--------------------------------

### Adjusting Session Timeout in a Notebook Environment

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Example code snippet to set the session timeout to 1 hour when using Sail in a notebook.

```python
import os

os.environ["SAIL_SPARK__SESSION_TIMEOUT_SECS"] = "3600"
```

--------------------------------

### Run PySpark Shell with Sail on Kubernetes

Source: https://docs.lakesail.com/sail/latest/guide/deployment/kubernetes.html

Example command to run a PySpark shell powered by Sail on Kubernetes, with worker pods launched on-demand.

```bash
env SPARK_CONNECT_MODE_ENABLED=1 SPARK_REMOTE="sc://localhost:50051" pyspark
```

--------------------------------

### Running the Code Formatter and Linter

Source: https://docs.lakesail.com/sail/latest/development/build/python.html

Formats the Python code using Ruff.

```bash
hatch fmt
```

--------------------------------

### Arrow UDTF Example

Source: https://docs.lakesail.com/sail/latest/guide/udf/examples.html

An example of defining and using an Arrow UDTF class in Python.

```python
>>> import pyarrow as pa
>>> import pyarrow.compute as pc
>>> from pyspark.sql.functions import arrow_udtf, lit
>>> 
>>> @arrow_udtf(returnType="v int")
... class Square:
...     def eval(self, x: pa.Array):
...         yield pa.table({"v": pc.multiply(x, x)})
... 
>>> Square(lit(3)).show()
+---+
|  v|
+---+
|  9|
+---+

```

--------------------------------

### Build Documentation Site

Source: https://docs.lakesail.com/sail/latest/development/documentation

Builds the documentation site using VitePress.

```bash
pnpm run docs:build
```

--------------------------------

### Installation

Source: https://docs.lakesail.com/sail/latest/guide/sources/vortex

Install the `pysail` package with the `vortex` extra to use the Vortex data source. The `vortex` extra installs the `vortex-data` Python library as a dependency.

```bash
pip install pysail[vortex]
```

--------------------------------

### Install dbt-sail

Source: https://docs.lakesail.com/sail/latest/guide/integrations/dbt.html

Command to install the dbt-sail package.

```bash
pip install dbt-sail
```

--------------------------------

### Installation

Source: https://docs.lakesail.com/sail/latest/guide/integrations/ibis.html

Install Ibis with the PySpark backend, along with a Spark Connect client.

```bash
pip install "ibis-framework[pyspark]>=12,<13"
```

--------------------------------

### Install target toolchain

Source: https://docs.lakesail.com/sail/latest/development/recipes/cross-compilation.html

Installs the specified target toolchain for cross-compilation.

```bash
rustup target add aarch64-unknown-linux-gnu
```

--------------------------------

### Build Sail from Source with CPU Optimization

Source: https://docs.lakesail.com/sail/latest/introduction/installation

Build Sail from source with all features of the current CPU enabled by setting RUSTFLAGS.

```bash
env RUSTFLAGS="-C target-cpu=native" pip install "pysail==0.6.3" -v --no-binary pysail
```

--------------------------------

### Unity Catalog Configuration

Source: https://docs.lakesail.com/sail/latest/guide/catalog/unity.html

Example of configuring Unity Catalog using environment variables and the catalog provider list.

```bash
export UNITY_ALLOW_HTTP_URL='false'
export SAIL_CATALOG__LIST='[{type="unity", name="sail", uri="https://catalog.example.com", default_catalog="meow", token="..."}]'
```

--------------------------------

### Python Example for Sail Compatibility

Source: https://docs.lakesail.com/sail/latest/reference/changelog

A Python example demonstrating how to report Sail compatibility for PySpark code.

```python
# This is a placeholder for the actual Python code example.
# The documentation does not contain explicit code blocks for this item.
# Example: 
# from sail import SailCompatibility
# 
# def check_pyspark_compatibility():
#     compatibility = SailCompatibility()
#     return compatibility.check_dataframe(spark.createDataFrame([(1, 'a')], ['col1', 'col2']))

```

--------------------------------

### Dockerfile

Source: https://docs.lakesail.com/sail/latest/guide/deployment/docker-images/quick-start.html

This Dockerfile is used to build the Sail Docker image from the Sail Python package.

```docker
FROM python:3.14-slim

ARG PYSAIL_VERSION
ARG PYSPARK_VERSION=4.1.1

RUN python3 -m pip install --no-cache-dir "pysail==${PYSAIL_VERSION}"
RUN python3 -m pip install --no-cache-dir "pyspark-client==${PYSPARK_VERSION}"

ENTRYPOINT ["/usr/local/bin/sail"]
```

--------------------------------

### Arrow Scalar UDFs - Basic Example

Source: https://docs.lakesail.com/sail/latest/guide/udf/examples.html

Defines and demonstrates an Arrow scalar UDF for squaring numbers.

```python
>>> import pyarrow as pa
>>> import pyarrow.compute as pc
>>> from pyspark.sql.functions import arrow_udf
>>> 
>>> @arrow_udf("long")
... def square(v: pa.Array) -> pa.Array:
...     return pc.multiply(v, v)
... 
>>> spark.range(3).select(square("id")).show()
+----------+
|square(id)|
+----------+
|         0|
|         1|
|         4|
+----------+

```

--------------------------------

### Memory Catalog Configuration

Source: https://docs.lakesail.com/sail/latest/guide/catalog/memory.html

An example of how to configure a memory catalog using environment variables.

```bash
export SAIL_CATALOG__LIST='[{type="memory", name="sail", initial_database=["dev", "analytics"], initial_database_comment="Development analytics database"}]'
```

--------------------------------

### Date Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/datetime.html

Examples demonstrating the usage of DATE literals in Databricks SQL queries.

```sql
SELECT DATE '2001' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2001-01-01|
-- +----------+

SELECT DATE '2005-07' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2005-07-01|
-- +----------+

SELECT DATE '2019-12-25' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2019-12-25|
-- +----------+
```

--------------------------------

### Example of NULL Literal

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/null.html

An example demonstrating the use of the NULL literal in a SELECT statement.

```sql
SELECT NULL AS col;
-- +----+
-- | col|
-- +----+
-- |NULL|
-- +----+
```

--------------------------------

### Python UDTFs - SQL Registration and Usage

Source: https://docs.lakesail.com/sail/latest/guide/udf/examples.html

Demonstrates registering a Python UDTF (`Tokenizer`) and using it within SQL queries.

```python
>>> from pyspark.sql.functions import udtf
>>> 
>>> @udtf(returnType="word: string")
... class Tokenizer:
...     def eval(self, text: str):
...         for word in text.split():
...             yield (word.strip().lower(),)
... 
>>> spark.udtf.register("tokenize", Tokenizer)
<...UserDefinedTableFunction object at ...>
>>> spark.sql("SELECT * FROM tokenize('Hello world')").show()
+-----+
| word|
+-----+
|hello|
|world|
+-----+

>>> spark.sql(
...     "SELECT id, word FROM VALUES (1, 'Hello world'), (2, 'Hi') AS t(id, text), "
...     "LATERAL tokenize(text) "
...     "ORDER BY id, word"
... ).show()
+---+-----+
| id| word|
+---+-----+
|  1|hello|
|  1|world|
|  2|   hi|
+---+-----+

```

--------------------------------

### Running Tests from the Installed Package

Source: https://docs.lakesail.com/sail/latest/development/build/python.html

Runs tests specifically from the installed package using `pytest`.

```bash
hatch run pytest --pyargs pysail
```