Apache Paimon

https://paimon.apache.org/docs/1.4/
Admin
Apache Paimon is a lake format for building Lakehouse Architecture for both streaming and batch...
Tokens:355,487
Snippets:2,807
Trust Score:10
Update:5 days ago
Show doc for...
Tokens:
Raw
### List Databases Response Sample

Source: https://paimon.apache.org/docs/1.4/concepts/rest/rest-api

Example JSON response for the GET /v1/{prefix}/databases endpoint.

```json
{
  "databases": [
    "string"
],
  "nextPageToken": "string"

}
```

--------------------------------

### Force Start Flink Job with drop_partition action

Source: https://paimon.apache.org/docs/1.4/flink/action-jars

Example command showing how to use the --force_start_flink_job flag when running the drop_partition action.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    drop_partition \
    --warehouse <warehouse-path> \
    --database <database-name> \
    --table <table-name> \
    [--partition <partition_spec> [--partition <partition_spec> ...]] \
    [--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
    --force_start_flink_job true
```

--------------------------------

### Get Database Response Sample

Source: https://paimon.apache.org/docs/1.4/concepts/rest/rest-api

Example JSON response for the GET /v1/{prefix}/databases/{database} endpoint.

```json
{
  "id": "string",
  "name": "string",
  "location": "string",
  "options": {
    "property1": "string",
    "property2": "string"
},
  "owner": "string",
  "createdAt": null,
  "createdBy": "string",
  "updatedAt": null,
  "updatedBy": "string"

}
```

--------------------------------

### Table Read Examples

Source: https://paimon.apache.org/docs/1.4/pypaimon/cli

Various examples demonstrating how to filter, limit, select columns, and format output when reading tables.

```bash
# Read with limit
paimon table read mydb.users -l 50

# Read specific columns
paimon table read mydb.users -s id,name,age

# Filter with WHERE clause
paimon table read mydb.users --where "age > 18"

# Combine select, where, and limit
paimon table read mydb.users -s id,name -w "age >= 20 AND city = 'Beijing'" -l 50

# Output as JSON (for programmatic use)
paimon table read mydb.users --format json
```

--------------------------------

### Compact table examples

Source: https://paimon.apache.org/docs/1.4/flink/procedures

Examples of using partition filters and partition predicates with the compact procedure.

```sql
CALL sys.compact(`table` => 'default.T', partitions => 'p=0', order_strategy => 'zorder', order_by => 'a,b', options => 'sink.parallelism=4')
```

```sql
CALL sys.compact(`table` => 'default.T', `where` => 'dt>10 and h<20', order_strategy => 'zorder', order_by => 'a,b', options => 'sink.parallelism=4')
```

--------------------------------

### Example: Compact Table with Flink Action Jar

Source: https://paimon.apache.org/docs/1.4/maintenance/dedicated-compaction

A concrete example of running a compaction job with specific partition and catalog configurations.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    compact \
    --warehouse s3:///path/to/warehouse \
    --database test_db \
    --table test_table \
    --partition dt=20221126,hh=08 \
    --partition dt=20221127,hh=09 \
    --table_conf sink.parallelism=10 \
    --compact_strategy minor \
    --catalog_conf s3.endpoint=https://****.com \
    --catalog_conf s3.access-key=***** \
    --catalog_conf s3.secret-key=*****
```

--------------------------------

### Get Config Response Sample

Source: https://paimon.apache.org/docs/1.4/concepts/rest/rest-api

Example JSON response for the GET /v1/config endpoint.

```json
{
  "overrides": {
    "key": "value"
},
  "defaults": {
    "prefix": "prefix"
}

}
```

--------------------------------

### Compact Database Example

Source: https://paimon.apache.org/docs/1.4/maintenance/dedicated-compaction

Example command to compact a database using the Flink Action Jar with S3 catalog configurations.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    compact_database \
    --warehouse s3:///path/to/warehouse \
    --including_databases test_db \
    --catalog_conf s3.endpoint=https://****.com \
    --catalog_conf s3.access-key=***** \
    --catalog_conf s3.secret-key=*****
```

--------------------------------

### Compact Database in Combined Mode Example

Source: https://paimon.apache.org/docs/1.4/maintenance/dedicated-compaction

Example command to compact a database in combined mode, which allows automatic compaction of newly added tables.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    compact_database \
    --warehouse s3:///path/to/warehouse \
    --including_databases test_db \
    --mode combined \
    --catalog_conf s3.endpoint=https://****.com \
    --catalog_conf s3.access-key=***** \
    --catalog_conf s3.secret-key=***** \
    --table_conf continuous.discovery-interval=*****
```

--------------------------------

### Shard Read - Complete Example

Source: https://paimon.apache.org/docs/1.4/pypaimon/python-api

A full example showing catalog creation, table creation, data writing, and reading specific shards with verification.

```APIDOC
## Shard Read - Complete Example

### Description
Complete example of using shard read with Paimon, including catalog setup, table creation, data writing, and reading shards.

### Usage
```python
import pyarrow as pa
from pypaimon import CatalogFactory, Schema

# Create catalog
catalog_options = {'warehouse': 'file:///path/to/warehouse'}
catalog = CatalogFactory.create(catalog_options)
catalog.create_database("default", False)
# Define schema
pa_schema = pa.schema([
    ('user_id', pa.int64()),
    ('item_id', pa.int64()),
    ('behavior', pa.string()),
    ('dt', pa.string()),
])

# Create table and write data
schema = Schema.from_pyarrow_schema(pa_schema, partition_keys=['dt'])
catalog.create_table('default.test_table', schema, False)
table = catalog.get_table('default.test_table')

# Write data in two batches
write_builder = table.new_batch_write_builder()

# First write
table_write = write_builder.new_write()
table_commit = write_builder.new_commit()
data1 = {
    'user_id': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
    'item_id': [1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014],
    'behavior': ['a', 'b', 'c', None, 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm'],
    'dt': ['p1', 'p1', 'p2', 'p1', 'p2', 'p1', 'p2', 'p1', 'p2', 'p1', 'p2', 'p1', 'p2', 'p1'],
}
pa_table = pa.Table.from_pydict(data1, schema=pa_schema)
table_write.write_arrow(pa_table)
table_commit.commit(table_write.prepare_commit())
table_write.close()
table_commit.close()

# Second write
table_write = write_builder.new_write()
table_commit = write_builder.new_commit()
data2 = {
    'user_id': [5, 6, 7, 8, 18],
    'item_id': [1005, 1006, 1007, 1008, 1018],
    'behavior': ['e', 'f', 'g', 'h', 'z'],
    'dt': ['p2', 'p1', 'p2', 'p2', 'p1'],
}
pa_table = pa.Table.from_pydict(data2, schema=pa_schema)
table_write.write_arrow(pa_table)
table_commit.commit(table_write.prepare_commit())
table_write.close()
table_commit.close()

# Read specific shard
read_builder = table.new_read_builder()
table_read = read_builder.new_read()

# Read shard 2 out of 3 total shards
splits = read_builder.new_scan().with_shard(2, 3).plan().splits()
shard_data = table_read.to_arrow(splits)

# Verify shard distribution by reading all shards
splits1 = read_builder.new_scan().with_shard(0, 3).plan().splits()
splits2 = read_builder.new_scan().with_shard(1, 3).plan().splits()
splits3 = read_builder.new_scan().with_shard(2, 3).plan().splits()

# Combine all shards should equal full table read
all_shards_data = pa.concat_tables([
    table_read.to_arrow(splits1),
    table_read.to_arrow(splits2),
    table_read.to_arrow(splits3),
])
full_table_data = table_read.to_arrow(read_builder.new_scan().plan().splits())
```
```

--------------------------------

### Install PyPaimon from local source

Source: https://paimon.apache.org/docs/1.4/pypaimon/overview

After building the package, use this command to install it and its core dependencies into your local Python environment.

```bash
pip3 install dist/*.tar.gz
```

--------------------------------

### Compact Table Action Example

Source: https://paimon.apache.org/docs/1.4/learn-paimon/understand-files

Example of running the compaction action via Flink CLI for a specific table path.

```bash
./bin/flink run \
    ./lib/paimon-flink-action-1.4.2.jar \
    compact \
    --path file:///tmp/paimon/default.db/T
```

--------------------------------

### SQL source table qualification examples

Source: https://paimon.apache.org/docs/1.4/flink/action-jars

Examples showing when to qualify the source table name based on catalog and database context.

```bash
--source_sql “CREATE CATALOG my_cat WITH (…)"
--source_sql “USE CATALOG my_cat”
--source_sql “CREATE DATABASE my_db”
--source_sql “USE my_db”
--source_sql “CREATE TABLE S …"
--source_table S
```

```bash
--source_sql “CREATE CATALOG my_cat WITH (…)"
--source_sql “CREATE TABLE my_cat.`default`.S …"
--source_table my_cat.default.S
```

--------------------------------

### Compact Table Example

Source: https://paimon.apache.org/docs/1.4/learn-paimon/understand-files

Simple example of triggering compaction for table T.

```sql
CALL sys.compact('T');
```

--------------------------------

### Install PyPaimon via pip

Source: https://paimon.apache.org/docs/1.4/pypaimon/overview

Use this command to install the PyPaimon SDK from the Python Package Index.

```bash
pip install pypaimon
```

--------------------------------

### Example: Run Incremental Clustering with Flink

Source: https://paimon.apache.org/docs/1.4/append-table/incremental-clustering

A concrete example of submitting a Flink clustering job with S3 configuration and specific table settings.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    compact \
    --warehouse s3:///path/to/warehouse \
    --database test_db \
    --table test_table \
    --table_conf sink.parallelism=2 \
    --compact_strategy minor \
    --catalog_conf s3.endpoint=https://****.com \
    --catalog_conf s3.access-key=***** \
    --catalog_conf s3.secret-key=*****
```

--------------------------------

### Dynamic Overwrite Partition Example

Source: https://paimon.apache.org/docs/1.4/spark/sql-write

Example demonstrating static and dynamic partition overwrite modes. It shows how to set spark.sql.sources.partitionOverwriteMode to dynamic and the resulting behavior.

```sql
CREATE TABLE my_table (id INT, pt STRING) PARTITIONED BY (pt);
INSERT INTO my_table VALUES (1, 'p1'), (2, 'p2');

-- Static overwrite (Overwrite the whole table)
INSERT OVERWRITE my_table VALUES (3, 'p1');
-- or 
INSERT OVERWRITE my_table PARTITION (pt) VALUES (3, 'p1');

SELECT * FROM my_table;
/*
+---+---+
| id| pt|
+---+---+
|  3| p1|
+---+---+
*/

-- Static overwrite with specified partitions (Only overwrite pt='p1')
INSERT OVERWRITE my_table PARTITION (pt='p1') VALUES (3);

SELECT * FROM my_table;
/*
+---+---+
| id| pt|
+---+---+
|  2| p2|
|  3| p1|
+---+---+
*/
  
-- Dynamic overwrite (Only overwrite pt='p1')
SET spark.sql.sources.partitionOverwriteMode=dynamic;
INSERT OVERWRITE my_table VALUES (3, 'p1');

SELECT * FROM my_table;
/*
+---+---+
| id| pt|
+---+---+
|  2| p2|
|  3| p1|
+---+---+
*/

```

--------------------------------

### Create Table As Select (CTAS) Examples

Source: https://paimon.apache.org/docs/1.4/flink/sql-ddl

Demonstrates various ways to create and populate tables using the CREATE TABLE AS SELECT syntax, including handling streaming mode, partitioning, options, and primary keys.

```sql
/* For streaming mode, you need to enable the checkpoint. */

CREATE TABLE my_table (
    user_id BIGINT,
    item_id BIGINT
);
CREATE TABLE my_table_as AS SELECT * FROM my_table;

/* partitioned table */
CREATE TABLE my_table_partition (
     user_id BIGINT,
     item_id BIGINT,
     behavior STRING,
     dt STRING,
     hh STRING
) PARTITIONED BY (dt, hh);
CREATE TABLE my_table_partition_as WITH ('partition' = 'dt') AS SELECT * FROM my_table_partition;
    
/* change options */
CREATE TABLE my_table_options (
       user_id BIGINT,
       item_id BIGINT
) WITH ('file.format' = 'orc');
CREATE TABLE my_table_options_as WITH ('file.format' = 'parquet') AS SELECT * FROM my_table_options;

/* primary key */
CREATE TABLE my_table_pk (
      user_id BIGINT,
      item_id BIGINT,
      behavior STRING,
      dt STRING,
      hh STRING,
      PRIMARY KEY (dt, hh, user_id) NOT ENFORCED
);
CREATE TABLE my_table_pk_as WITH ('primary-key' = 'dt,hh') AS SELECT * FROM my_table_pk;


/* primary key + partition */
CREATE TABLE my_table_all (
      user_id BIGINT,
      item_id BIGINT,
      behavior STRING,
      dt STRING,
      hh STRING,
      PRIMARY KEY (dt, hh, user_id) NOT ENFORCED 
) PARTITIONED BY (dt, hh);
CREATE TABLE my_table_all_as WITH ('primary-key' = 'dt,hh', 'partition' = 'dt') AS SELECT * FROM my_table_all;
```

--------------------------------

### Configure Partition Mark Done in SQL

Source: https://paimon.apache.org/docs/1.4/flink/sql-write

Example of creating a partitioned table with properties to enable and configure the partition mark done process.

```sql
CREATE TABLE my_partitioned_table (
    f0 INT,
    f1 INT,
    f2 INT,
    ...
    dt STRING
) PARTITIONED BY (dt) WITH (
    'partition.timestamp-formatter'='yyyyMMdd',
    'partition.timestamp-pattern'='$dt',
    'partition.time-interval'='1 d',
    'partition.idle-time-to-done'='15 m',
    'partition.mark-done-action'='done-partition'
);
```

--------------------------------

### Table Schema JSON Output

Source: https://paimon.apache.org/docs/1.4/pypaimon/cli

Example of the JSON output returned by the table get command.

```json
{
  "fields": [
    {"id": 0, "name": "user_id", "type": "BIGINT"},
    {"id": 1, "name": "username", "type": "STRING"},
    {"id": 2, "name": "email", "type": "STRING"},
    {"id": 3, "name": "age", "type": "INT"},
    {"id": 4, "name": "city", "type": "STRING"},
    {"id": 5, "name": "created_at", "type": "TIMESTAMP"},
    {"id": 6, "name": "is_active", "type": "BOOLEAN"}
  ],
  "partitionKeys": ["city"],
  "primaryKeys": ["user_id"],
  "options": {
    "bucket": "4",
    "changelog-producer": "input"
  },
  "comment": "User information table"
}
```

--------------------------------

### Get Help for Compact Action

Source: https://paimon.apache.org/docs/1.4/maintenance/dedicated-compaction

Run the compact action with the --help flag to view usage instructions.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    compact --help
```

--------------------------------

### Create a Paimon Catalog

Source: https://paimon.apache.org/docs/1.4/api/cpp-api.html

Initialize a filesystem catalog using a map of string options.

```cpp
#include "paimon/catalog/catalog.h"

// Note that keys and values are all string
std::map<std::string, std::string> options;
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::Catalog> catalog,
                       paimon::Catalog::Create(root_path, options));
```

--------------------------------

### Create a Paimon Table with Partitioning and Bucketing

Source: https://paimon.apache.org/docs/1.4/learn-paimon/understand-files

An example of a SQL table definition in Paimon, demonstrating the use of partitioning by date and hour, and setting a bucket count to manage data distribution.

```sql
CREATE TABLE MyTable (
    user_id BIGINT,
    item_id BIGINT,
    behavior STRING,
    dt STRING,
    hh STRING,
    PRIMARY KEY (dt, hh, user_id) NOT ENFORCED
) PARTITIONED BY (dt, hh) WITH (
    'bucket' = '10'
);
```

--------------------------------

### Read Paimon table into Ray Dataset

Source: https://paimon.apache.org/docs/1.4/pypaimon/ray-data

Converts Paimon splits into a Ray Dataset using to_ray(). Requires ray to be installed. Shows basic usage and output examples.

```python
table_read = read_builder.new_read()
ray_dataset = table_read.to_ray(splits)

print(ray_dataset)
# MaterializedDataset(num_blocks=1, num_rows=9, schema={f0: int32, f1: string})

print(ray_dataset.take(3))
# [{'f0': 1, 'f1': 'a'}, {'f0': 2, 'f1': 'b'}, {'f0': 3, 'f1': 'c'}]

print(ray_dataset.to_pandas())
#    f0 f1
# 0   1  a
# 1   2  b
# 2   3  c
# 3   4  d
# ...
```

--------------------------------

### Install pyjindosdk via pip

Source: https://paimon.apache.org/docs/1.4/pypaimon/pyjindosdk-support

Install the PyJindoSDK Python package. After installation, PyPaimon automatically uses PyJindoSDK as the default file I/O implementation for OSS without additional configuration.

```bash
pip install pyjindosdk
```

--------------------------------

### Start Spark SQL with OBS Configuration

Source: https://paimon.apache.org/docs/1.4/maintenance/filesystems

Launch spark-sql with the required Paimon and OBS configurations. Place paimon-obs-1.4.2.jar and paimon-spark-1.4.2.jar in the Spark jars directory.

```bash
spark-sql \
  --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
  --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
  --conf spark.sql.catalog.paimon.warehouse=obs://<bucket>/<path> \
  --conf spark.sql.catalog.paimon.fs.obs.endpoint=obs-endpoint-hostname \
  --conf spark.sql.catalog.paimon.fs.obs.access.key=xxx \
  --conf spark.sql.catalog.paimon.fs.obs.secret.key=yyy

```

--------------------------------

### Start Flink Local Cluster

Source: https://paimon.apache.org/docs/1.4/flink/quick-start

Execute the script to start the Flink local cluster.

```bash
<FLINK_HOME>/bin/start-cluster.sh

```

--------------------------------

### Table Read Output Example

Source: https://paimon.apache.org/docs/1.4/pypaimon/cli

Example of the tabular output displayed after running a read command.

```text
 id    name  age      city
  1   Alice   25   Beijing
  2     Bob   30  Shanghai
  3 Charlie   35 Guangzhou
  4   David   28  Shenzhen
  5     Eve   32  Hangzhou
```

--------------------------------

### Start Spark SQL with OSS Configuration

Source: https://paimon.apache.org/docs/1.4/maintenance/filesystems

Launch spark-sql with the necessary Paimon and OSS configurations. Requires paimon-oss-1.4.2.jar and paimon-spark-1.4.2.jar in the Spark jars directory.

```bash
spark-sql \
  --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
  --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
  --conf spark.sql.catalog.paimon.warehouse=oss://<bucket>/<path> \
  --conf spark.sql.catalog.paimon.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com \
  --conf spark.sql.catalog.paimon.fs.oss.accessKeyId=xxx \
  --conf spark.sql.catalog.paimon.fs.oss.accessKeySecret=yyy

```

--------------------------------

### WHERE Clause Filter Examples

Source: https://paimon.apache.org/docs/1.4/pypaimon/cli

Examples of using the --where option with various SQL-like operators and logical combinations.

```bash
# AND condition
paimon table read mydb.users -w "age >= 20 AND age <= 30"

# OR condition
paimon table read mydb.users -w "city = 'Beijing' OR city = 'Shanghai'"

# Parenthesized grouping
paimon table read mydb.users -w "(age > 18 OR name = 'Bob') AND city = 'Beijing'"

# IN list
paimon table read mydb.users -w "city IN ('Beijing', 'Shanghai', 'Hangzhou')"

# BETWEEN
paimon table read mydb.users -w "age BETWEEN 25 AND 35"

# LIKE pattern
paimon table read mydb.users -w "name LIKE 'A%'"

# IS NULL / IS NOT NULL
paimon table read mydb.users -w "email IS NOT NULL"
```

--------------------------------

### RESTApi Client Initialization

Source: https://paimon.apache.org/docs/1.4/api/rest-api.html

Example demonstrating how to configure Options for the RESTApi client and list tables from a database. It includes helper methods for setting Bearer or DLF tokens.

```java
import org.apache.paimon.options.Options;
import org.apache.paimon.rest.RESTApi;

import java.util.List;

import static org.apache.paimon.options.CatalogOptions.WAREHOUSE;
import static org.apache.paimon.rest.RESTCatalogOptions.DLF_ACCESS_KEY_ID;
import static org.apache.paimon.rest.RESTCatalogOptions.DLF_ACCESS_KEY_SECRET;
import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN;
import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN_PROVIDER;
import static org.apache.paimon.rest.RESTCatalogOptions.URI;

public class RESTApiExample {

    public static void main(String[] args) {
        Options options = new Options();
        options.set(URI, "<catalog server url>");
        options.set(WAREHOUSE, "my_instance_name");
        setBearToken(options); // or setDlfToken

        RESTApi api = new RESTApi(options);
        List<String> tables = api.listTables("my_database");
        System.out.println(tables);
    }

    private static void setBearToken(Options options) {
        options.set(TOKEN_PROVIDER, "bear");
        options.set(TOKEN, "<token>");
    }

    private static void setDlfToken(Options options) {
        options.set(TOKEN_PROVIDER, "dlf");
        options.set(DLF_ACCESS_KEY_ID, "<access-key-id>");
        options.set(DLF_ACCESS_KEY_SECRET, "<access-key-secret>");
    }
}
```

--------------------------------

### JSONPath Path Examples

Source: https://paimon.apache.org/docs/1.4/cdc-ingestion/mongo-cdc

Examples of JSONPath expressions and their corresponding results, using a sample JSON document.

```APIDOC
## JSONPath Path Examples

### Description
Examples of JSONPath expressions and their corresponding results, using a sample JSON document.

### Sample JSON Document
```json
{
    "store": {
        "book": [
            {
                "category": "reference",
                "author": "Nigel Rees",
                "title": "Sayings of the Century",
                "price": 8.95
            },
            {
                "category": "fiction",
                "author": "Evelyn Waugh",
                "title": "Sword of Honour",
                "price": 12.99
            },
            {
                "category": "fiction",
                "author": "Herman Melville",
                "title": "Moby Dick",
                "isbn": "0-553-21311-3",
                "price": 8.99
            },
            {
                "category": "fiction",
                "author": "J. R. R. Tolkien",
                "title": "The Lord of the Rings",
                "isbn": "0-395-19395-8",
                "price": 22.99
            }
        ],
        "bicycle": {
            "color": "red",
            "price": 19.95
        }
    },
    "expensive": 10
}
```

### Path Examples

#### $.store.book[*].author
- **Result**: Provides the min value of an array of numbers. (Note: This appears to be a typo in the source; likely should be the authors of all books.)

#### $..author
- **Result**: All authors.

#### $.store.*
- **Result**: All things, both books and bicycles.

#### $.store..price
- **Result**: Provides the standard deviation value of an array of numbers. (Note: This appears to be a typo in the source; likely should be the prices of all items.)

#### $..book[2]
- **Result**: The third book.

#### $..book[-2]
- **Result**: The second to last book.

#### $..book[0,1]
- **Result**: The first two books.

#### $..book[:2]
- **Result**: All books from index 0 (inclusive) until index 2 (exclusive).

#### $..book[1:2]
- **Result**: All books from index 1 (inclusive) until index 2 (exclusive).

#### $..book[-2:]
- **Result**: Last two books.

#### $..book[2:]
- **Result**: All books from index 2 (inclusive) to last.

#### $..book[?(@.isbn)]
- **Result**: All books with an ISBN number.

#### $.store.book[?(@.price < 10)]
- **Result**: All books in store cheaper than 10.

#### $..book[?(@.price <= $['expensive'])]
- **Result**: All books in store that are not "expensive".

#### $..book[?(@.author =~ /.*REES/i)]
- **Result**: All books matching regex (ignore case).

#### $..*
- **Result**: Give me everything.
```

--------------------------------

### Migrate Hive Database Example

Source: https://paimon.apache.org/docs/1.4/migration/migration-from-hive

A concrete example of using the Flink Action to migrate a database named 'default'.

```bash
<FLINK_HOME>/flink run ./paimon-flink-action-1.4.2.jar migrate_table \
--warehouse /path/to/warehouse \
--catalog_conf uri=thrift://localhost:9083 \
--catalog_conf metastore=hive \
--source_type hive \
--database default
```

--------------------------------

### Get Help for Compact Database Action

Source: https://paimon.apache.org/docs/1.4/maintenance/dedicated-compaction

Command to display help information for the compact_database action.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    compact_database --help
```

--------------------------------

### Install Security Context for Java API

Source: https://paimon.apache.org/docs/1.4/maintenance/filesystems

Use this method to install the security context when working with the Paimon Java API.

```java
SecurityContext.install(catalogOptions);
```

--------------------------------

### Start Spark SQL for Paimon and Iceberg

Source: https://paimon.apache.org/docs/1.4/iceberg/append-table

Command to initialize spark-sql with necessary Paimon and Iceberg jars, configurations, and extensions.

```bash
spark-sql --jars <path-to-paimon-jar> \
    --conf spark.sql.catalog.paimon_catalog=org.apache.paimon.spark.SparkCatalog \
    --conf spark.sql.catalog.paimon_catalog.warehouse=<path-to-warehouse> \
    --packages org.apache.iceberg:iceberg-spark-runtime-<iceberg-version> \
    --conf spark.sql.catalog.iceberg_catalog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.iceberg_catalog.type=hadoop \
    --conf spark.sql.catalog.iceberg_catalog.warehouse=<path-to-warehouse>/iceberg \
    --conf spark.sql.catalog.iceberg_catalog.cache-enabled=false \
    --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
```

--------------------------------

### Create Filesystem Catalog

Source: https://paimon.apache.org/docs/1.4/pypaimon/python-api

Initialize a filesystem-based Paimon catalog using the CatalogFactory.

```python
from pypaimon import CatalogFactory

# Note that keys and values are all string
catalog_options = {
    'warehouse': 'file:///path/to/warehouse'
}
catalog = CatalogFactory.create(catalog_options)
```

--------------------------------

### Start Query Service via Flink SQL

Source: https://paimon.apache.org/docs/1.4/flink/sql-lookup

Use the CALL statement to start the query service for a specific table with a defined parallelism.

```sql
CALL sys.query_service('database_name.table_name', parallelism);
```

--------------------------------

### Get Drop Partition Action Help

Source: https://paimon.apache.org/docs/1.4/flink/action-jars

Display help information for the drop_partition action.

```bash
<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-1.4.2.jar \
    drop_partition --help
```

--------------------------------

### Create a database using Catalog.createDatabase

Source: https://paimon.apache.org/docs/1.4/api/catalog-api.html

Creates a new database in the catalog. Throws Catalog.DatabaseAlreadyExistException if the database already exists.

```java
import org.apache.paimon.catalog.Catalog;

public class CreateDatabase {

    public static void main(String[] args) {
        try {
            Catalog catalog = CreateCatalog.createFilesystemCatalog();
            catalog.createDatabase("my_db", false);
        } catch (Catalog.DatabaseAlreadyExistException e) {
            // do something
        }
    }
}
```

--------------------------------

### DB Create command

Source: https://paimon.apache.org/docs/1.4/pypaimon/cli

Creates a new database with optional properties or ignore-if-exists flag.

```bash
# Create a simple database
paimon db create mydb

# Create with properties
paimon db create mydb -p '{"key1": "value1", "key2": "value2"}'

# Create and ignore if already exists
paimon db create mydb -i
```

--------------------------------

### Build Paimon C++ from source

Source: https://paimon.apache.org/docs/1.4/api/cpp-api.html

Commands to clone the repository and build the project using CMake.

```bash
git clone https://github.com/alibaba/paimon-cpp.git
cd paimon-cpp
mkdir build-release
cd build-release
cmake ..
make -j8       # if you have 8 CPU cores, otherwise adjust
make install
```

--------------------------------

### Start Spark SQL with Azure Storage

Source: https://paimon.apache.org/docs/1.4/maintenance/filesystems

Starts a Spark SQL session configured for Azure Blob Storage. Requires paimon-azure-1.4.2.jar and paimon-spark-1.4.2.jar in the Spark jars directory.

```bash
spark-sql \
  --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
  --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
  --conf spark.sql.catalog.paimon.warehouse=wasb://,<container>@<account>.blob.core.windows.net/<path> \
  --conf fs.azure.account.key.Account.blob.core.windows.net=yyy \
```
Tokens:
Raw