Sail

https://docs.lakesail.com/sail/latest/
Admin
Sail is an open-source multimodal compute engine for batch processing, stream processing, and...
Tokens:95,039
Snippets:836
Trust Score:7.6
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Basic Vortex read with show

Source: https://docs.lakesail.com/sail/latest/guide/sources/vortex

Basic example that registers the data source, reads a Vortex file, and displays the DataFrame. This is the minimal setup for reading Vortex data.

```python
from pysail.spark.datasource.vortex import VortexDataSource

spark.dataSource.register(VortexDataSource)

df = spark.read.format("vortex").option("path", "/data/events.vortex").load()
df.show()
```

--------------------------------

### Install Spark-to-Sail scanner with pip

Source: https://docs.lakesail.com/sail/latest/introduction/migrating-from-spark

Installs the Spark-to-Sail scanner CLI using pip. Alternative to the uv installation command.

```bash
pip install https://migrate.lakesail.com/dist/spark_to_sail-0.1.0-py3-none-any.whl
```

--------------------------------

### Create Dockerfile for Sail quick start

Source: https://docs.lakesail.com/sail/latest/guide/deployment/docker-images/quick-start.html

Creates a Dockerfile that installs the Sail Python package and a compatible PySpark client. Requires setting the PYSAIL_VERSION build argument and optionally PYSPARK_VERSION. Not suitable for production deployments.

```docker
ARG PYTHON_VERSION=3.14

FROM python:${PYTHON_VERSION}-slim

ARG PYSAIL_VERSION
ARG PYSPARK_VERSION=4.2.0

RUN python3 -m pip install --no-cache-dir "pysail==${PYSAIL_VERSION}"
RUN python3 -m pip install --no-cache-dir "pyspark-client==${PYSPARK_VERSION}"

ENTRYPOINT ["/usr/local/bin/sail"]
```

--------------------------------

### Configure memory catalogs with environment variables

Source: https://docs.lakesail.com/sail/latest/guide/catalog

Sets the catalog list and default catalog via environment variables before starting the Sail server. The example configures two catalogs: a memory catalog named 'c1' with an initial database 'default', and an Iceberg REST catalog named 'c2' with a URI.

```bash
export SAIL_CATALOG__LIST='[{name="c1", type="memory", initial_database=["default"]}, {name="c2", type="iceberg-rest", uri="https://catalog.example.com"}]'
export SAIL_CATALOG__DEFAULT_CATALOG="c1"
```

--------------------------------

### Install Sail from PyPI

Source: https://docs.lakesail.com/sail/latest/introduction/installation

Installs Sail version 0.7.2 from PyPI using pip.

```bash
pip install "pysail==0.7.2"
```

--------------------------------

### List all sessions using spark.sql

Source: https://docs.lakesail.com/sail/latest/guide/catalog/system

Query the system catalog to list all sessions. Requires a Spark client session connected to the Sail server, as described in the Getting Started guide.

```python
spark.sql("SELECT * FROM system.session.sessions").show()
```

--------------------------------

### Start a Spark Connect server with sail spark server

Source: https://docs.lakesail.com/sail/latest/guide/cli

Start a Spark Connect server that uses Sail for computation, listening on the specified IP and port.

```bash
sail spark server --ip 127.0.0.1 --port 50051
```

--------------------------------

### Integral Number Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/numeric.html

Shows integral literals with and without postfixes. The examples demonstrate INT (no postfix), BIGINT (L), TINYINT (Y), and SMALLINT (S) literals, each with its corresponding output table.

```sql
SELECT -2147483600 AS col;
-- +-----------+
-- |        col|
-- +-----------+
-- |-2147483600|
-- +-----------+

SELECT 9223372036854775807L AS col;
-- +-------------------+
-- |                col|
-- +-------------------+
-- |9223372036854775807|
-- +-------------------+

SELECT -64Y AS col;
-- +----+
-- | col|
-- +----+
-- |-64 |
-- +----+

SELECT 512S AS col;
-- +----+
-- |col |
-- +----+
-- |512 |
-- +----+
```

--------------------------------

### Date literal examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/datetime.html

Examples of date literals with year, year-month, and full date. Each shows the resulting column output.

```sql
SELECT DATE '2001' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2001-01-01|
-- +----------+

SELECT DATE '2005-07' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2005-07-01|
-- +----------+

SELECT DATE '2019-12-25' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |2019-12-25|
-- +----------+
```

--------------------------------

### Multi-unit Interval Literal Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/interval.html

Shows four multi-unit interval literal examples with their expected outputs, demonstrating combinations of weeks, days, hours, minutes, seconds, milliseconds, and microseconds.

```sql
SELECT INTERVAL 3 WEEK AS col;
-- +-----------------+
-- |col              |
-- +-----------------+
-- |INTERVAL '21' DAY|
-- +-----------------+

SELECT INTERVAL -2 WEEKS '3' DAYS AS col;
-- +------------------+
-- |col               |
-- +------------------+
-- |INTERVAL '-11' DAY|
-- +------------------+

SELECT INTERVAL '3 DAYS 50 SECONDS' AS col;
-- +-----------------------------------+
-- |col                                |
-- +-----------------------------------+
-- |INTERVAL '3 00:00:50' DAY TO SECOND|
-- +-----------------------------------+

SELECT INTERVAL 3 WEEK 4 DAYS 5 HOUR 6 MINUTES 7 SECOND 8 MILLISECOND 9 MICROSECONDS AS col;
-- +-------------------------------------------+
-- |col                                        |
-- +-------------------------------------------+
-- |INTERVAL '25 05:06:07.008009' DAY TO SECOND|
-- +-------------------------------------------+
```

--------------------------------

### Install Ibis with PySpark backend

Source: https://docs.lakesail.com/sail/latest/guide/integrations/ibis.html

Install Ibis with the PySpark backend and a Spark Connect client. Requires Python and pip.

```bash
pip install "ibis-framework[pyspark]>=12,<13"
```

--------------------------------

### Authenticate with Kerberos and start Sail server

Source: https://docs.lakesail.com/sail/latest/guide/catalog/hms.html

Shows how to authenticate with Kerberos using kinit and then start the Sail server. In a distributed deployment, every worker needs its own Kerberos credentials.

```python
# authenticate with Kerberos
subprocess.run([
    "kinit",
    "-kt",
    "/path/to/user.keytab",
    "username@YOUR.REALM",
], check=True)

# start the Sail server
server = SparkConnectServer(ip="0.0.0.0", port=50051)
server.start(background=False)
```

--------------------------------

### Install Spark-to-Sail scanner with uv

Source: https://docs.lakesail.com/sail/latest/introduction/migrating-from-spark

Installs the Spark-to-Sail scanner CLI using uv. The scanner reports which Spark features your code uses and whether Sail supports them.

```bash
uv tool install https://migrate.lakesail.com/dist/spark_to_sail-0.1.0-py3-none-any.whl
```

--------------------------------

### Install langchain-sail with pip

Source: https://docs.lakesail.com/sail/latest/guide/integrations/langchain.html

Installs langchain-sail with the Spark Connect client, LangChain, and the OpenAI integration. The spark extra installs pyspark-client==4.2.0 without JVM components.

```bash
pip install "langchain-sail[spark]" langchain langchain-openai
```

--------------------------------

### Start Sail server with Kerberos ticket cache

Source: https://docs.lakesail.com/sail/latest/guide/catalog/hms.html

Run kinit before starting the Sail server so the server process can use its ticket cache.

```python
import subprocess
from pysail.spark import SparkConnectServer

```

--------------------------------

### Fractional Number Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/numeric.html

Shows fractional literals with and without postfixes. The examples demonstrate decimal (no postfix), double (E or D), decimal with BD, and float with F, each with its corresponding output table.

```sql
SELECT 76.543 AS col, TYPEOF(76.543) AS type;
-- +-------+------------+
-- |    col|        type|
-- +-------+------------+
-- | 76.543|decimal(5,3)|
-- +-------+------------+

SELECT 8.21E1 AS col, TYPEOF(8.21E1) AS type;
-- +------+------+
-- |   col|  type|
-- +------+------+
-- | 82.1 |double|
-- +------+------+

SELECT -0.4321 AS col;
-- +--------+
-- |     col|
-- +--------+
-- |-0.4321 |
-- +--------+

SELECT 250.BD AS col;
-- +-----+
-- | col |
-- +-----+
-- |250  |
-- +-----+

SELECT 6.9D AS col;
-- +-----+
-- | col |
-- +-----+
-- | 6.9 |
-- +-----+

SELECT -18BD AS col;
-- +----+
-- |col |
-- +----+
-- |-18 |
-- +----+

SELECT .789E3F AS col;
-- +------+
-- | col  |
-- +------+
-- |789.0 |
-- +------+
```

--------------------------------

### Install pysail with jdbc extra

Source: https://docs.lakesail.com/sail/latest/guide/sources/jdbc

Install the pysail package with the jdbc extra to use the JDBC data source.

```bash
pip install pysail[jdbc]
```

--------------------------------

### ANSI Interval Literal Examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/interval.html

Shows two ANSI interval literal examples: one for YEAR TO MONTH and one for DAY TO SECOND, each with its expected output.

```sql
SELECT INTERVAL '5-6' YEAR TO MONTH AS col;
-- +----------------------------+
-- |col                         |
-- +----------------------------+
-- |INTERVAL '5-6' YEAR TO MONTH|
-- +----------------------------+

SELECT INTERVAL -'12 23:45:59.888888' DAY TO SECOND AS col;
-- +--------------------------------------------+
-- |col                                         |
-- +--------------------------------------------+
-- |INTERVAL '-12 23:45:59.888888' DAY TO SECOND|
-- +--------------------------------------------+
```

--------------------------------

### Install langchain-sail with uv

Source: https://docs.lakesail.com/sail/latest/guide/integrations/langchain.html

Adds langchain-sail with the Spark Connect client, LangChain, and the OpenAI integration using uv. The spark extra installs pyspark-client==4.2.0 without JVM components.

```bash
uv add "langchain-sail[spark]" langchain langchain-openai
```

--------------------------------

### Start an interactive PySpark shell with sail spark shell

Source: https://docs.lakesail.com/sail/latest/guide/cli

Start an interactive PySpark shell that uses Sail for computation.

```bash
sail spark shell
```

--------------------------------

### Start Sail locally and connect

Source: https://docs.lakesail.com/sail/latest/guide/integrations/langchain.html

Starts a local Sail server in the application process and connects a Spark session to its listening address. Keep the server running while the integration is in use; stop the Spark session with spark.stop() before calling server.stop().

```python
from pysail.spark import SparkConnectServer
from pyspark.sql import SparkSession

server = SparkConnectServer()
server.start()
_, port = server.listening_address

spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate()
```

--------------------------------

### Binary literal example with output

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/binary.html

Example SELECT query using a binary literal X'123456' and its result set showing the byte sequence [12 34 56].

```sql
SELECT X'123456' AS col;
-- +----------+
-- |       col|
-- +----------+
-- |[12 34 56]|
-- +----------+
```

--------------------------------

### Create SailSQLToolkit and list tools

Source: https://docs.lakesail.com/sail/latest/guide/integrations/langchain.html

Initializes a chat model, a SailSQL instance with sample_rows_in_table_info=3, and a SailSQLToolkit. Prints the names of the tools in the toolkit.

```python
from langchain.chat_models import init_chat_model
from langchain_sail import SailSQL, SailSQLToolkit

model = init_chat_model("openai:gpt-5.5")

sail_sql = SailSQL(
    spark_session=spark,
    sample_rows_in_table_info=3,
)

toolkit = SailSQLToolkit(db=sail_sql, llm=model)
tools = toolkit.get_tools()

print([tool.name for tool in tools])
```

--------------------------------

### Configure Unity Catalog provider with environment variables

Source: https://docs.lakesail.com/sail/latest/guide/catalog/unity.html

Configures a Unity Catalog provider via environment variables. Sets UNITY_ALLOW_HTTP_URL to false and defines a catalog list with type unity, name sail, URI https://catalog.example.com, default_catalog meow, and a token placeholder.

```bash
export UNITY_ALLOW_HTTP_URL='false'
export SAIL_CATALOG__LIST='[{type="unity", name="sail", uri="https://catalog.example.com", default_catalog="meow", token="..."}]'

```

--------------------------------

### Install dbt-sail with pip

Source: https://docs.lakesail.com/sail/latest/guide/integrations/dbt.html

Installs the dbt-sail adapter, which pulls in dbt-spark[session] and pysail as dependencies. Sail does not need to be installed separately.

```bash
pip install dbt-sail
```

--------------------------------

### Timestamp literal examples

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/datetime.html

Examples of timestamp literals with fractional seconds, time zone offset, and partial date. The UTC-05:00 example shows conversion to UTC (17:00:00).

```sql
SELECT TIMESTAMP '2000-02-29 23:59:59.123' AS col;
-- +-----------------------+
-- |                    col|
-- +-----------------------+
-- |2000-02-29 23:59:59.123|
-- +-----------------------+

SELECT TIMESTAMP '2015-06-30 12:00:00.999999UTC-05:00' AS col;
-- +--------------------------+
-- |                      col |
-- +--------------------------+
-- |2015-06-30 17:00:00.999999|
-- +--------------------------+

SELECT TIMESTAMP '2010-08' AS col;
-- +-------------------+
-- |                col|
-- +-------------------+
-- |2010-08-01 00:00:00|
-- +-------------------+
```

--------------------------------

### Start Sail PySpark shell

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Start a PySpark shell from Sail. The SparkSession instance is available as the spark local variable and communicates with the Sail server started in the same Python interpreter process.

```bash
sail spark shell

```

--------------------------------

### Install PySail with pyspark[connect] for Spark 4.2

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Install PySail and PySpark with Spark Connect extras for Spark 4.2. The pyspark[connect] extra installs dependencies needed for Spark Connect.

```bash
pip install "pysail==0.7.2"
pip install "pyspark[connect]==4.2.0"

```

--------------------------------

### Install pysail with pip

Source: https://docs.lakesail.com/sail/latest/guide/integrations/langchain.html

Installs pysail to run a local Sail server for development or testing.

```bash
pip install pysail
```

--------------------------------

### Install pysail with vortex extra

Source: https://docs.lakesail.com/sail/latest/guide/sources/vortex

Install the pysail package with the vortex extra to use the Vortex data source. The vortex extra installs the vortex-data Python library as a dependency. Note that vortex-data is not yet available on Windows.

```bash
pip install pysail[vortex]
```

--------------------------------

### Create, insert, and query a table with Spark SQL using memory://

Source: https://docs.lakesail.com/sail/latest/guide/storage/memory.html

Uses Spark SQL to create a table backed by the in-memory storage, insert rows, and query the table. The `LOCATION 'memory:///users'` clause points to the temporary RAM storage, and the example shows the initial empty table and the result after inserting two rows.

```python
sql = """
CREATE TABLE my_table (id INT, name STRING)
USING parquet
LOCATION 'memory:///users'
"""
spark.sql(sql)
spark.sql("SELECT * FROM my_table").show()

spark.sql("INSERT INTO my_table VALUES (3, 'Charlie'), (4, 'David')")
spark.sql("SELECT * FROM my_table").show()
```

--------------------------------

### Configure MinIO with environment variables

Source: https://docs.lakesail.com/sail/latest/guide/storage/s3.html

Configure MinIO endpoint and credentials via environment variables.

```bash
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_ENDPOINT="http://localhost:9000"
```

--------------------------------

### Install pysail with uv

Source: https://docs.lakesail.com/sail/latest/guide/integrations/langchain.html

Adds pysail using uv to run a local Sail server for development or testing.

```bash
uv add pysail
```

--------------------------------

### Example: SELECT NULL AS col

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/null.html

Example query that selects a null literal as column 'col'. The output shows the result table with a NULL value.

```sql
SELECT NULL AS col;
-- +----+
-- | col|
-- +----+
-- |NULL|
-- +----+
```

--------------------------------

### Use Sail library to start Spark Connect server and run queries

Source: https://docs.lakesail.com/sail/latest/introduction/getting-started

Start a Spark Connect server using the Sail library and connect to it with PySpark. Demonstrates running a simple query, reading multimodal data with the DataFrame API, and creating a table over binary files in object storage.

```python
from pysail.spark import SparkConnectServer
from pyspark.sql import SparkSession

server = SparkConnectServer()
server.start()
_, port = server.listening_address

spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate()

# Run a simple query
spark.sql("SELECT 1 + 1").show()

# Use the DataFrame API to read multimodal data
spark.read.format("binaryFile").option("pathGlobFilter", "*.png").load("/path/to/data").show()

# Use Spark SQL to create a table that refers to multimodal data in an object storage
spark.sql("""
CREATE TABLE pdfs
USING binaryFile
OPTIONS (pathGlobFilter '*.pdf')
LOCATION 's3://my-bucket/path/to/data'
""")
spark.sql("SELECT * FROM pdfs").show()

```

--------------------------------

### Boolean literal examples: SELECT TRUE/FALSE

Source: https://docs.lakesail.com/sail/latest/guide/sql/literals/boolean.html

Example queries showing the output of selecting TRUE and FALSE as a column. The result displays 'true' and 'false' in lowercase.

```sql
SELECT TRUE AS col;
-- +----+
-- | col|
-- +----+
-- |true|
-- +----+

SELECT FALSE AS col;
-- +-----+
-- |  col|
-- +-----+
-- |false|
-- +-----+
```

--------------------------------

### Map tool names to instances

Source: https://docs.lakesail.com/sail/latest/guide/integrations/langchain.html

Builds a dictionary mapping tool names to tool instances so tools can be invoked directly by name without an agent.

```python
tools_by_name = {tool.name: tool for tool in tools}
```
Tokens:
Raw