Apache Iceberg

https://iceberg.apache.org/docs/latest/
Admin
Apache Iceberg is an open table format for huge analytic datasets that provides high-performance...
Tokens:153,164
Snippets:1,282
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Start Spark SQL shell with S3 Analytics Accelerator enabled

Source: https://iceberg.apache.org/docs/latest/aws

Use this command to launch Spark SQL with the Iceberg catalog configured for S3 Analytics Accelerator. Set `s3.analytics-accelerator.enabled=true` to enable the library; the default is `false`. This example uses the Glue catalog and S3FileIO.

```bash
spark-sql --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket2/my/key/prefix \
    --conf spark.sql.catalog.my_catalog.type=glue \
    --conf spark.sql.catalog.my_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
    --conf spark.sql.catalog.my_catalog.s3.analytics-accelerator.enabled=true

```

--------------------------------

### Create catalog through YAML config

Source: https://iceberg.apache.org/docs/latest/flink-ddl

Registers a catalog in 'sql-client-defaults.yaml' before starting the SQL client. The example defines a Hadoop catalog named 'my_catalog'.

```yaml
catalogs: 
  - name: my_catalog
    type: iceberg
    catalog-type: hadoop
    warehouse: hdfs://nn:8020/warehouse/path
```

--------------------------------

### Start Standalone Flink Cluster

Source: https://iceberg.apache.org/docs/latest/flink

Commands to set up the Hadoop environment and start a standalone Flink cluster.

```bash
# HADOOP_HOME is your hadoop root directory after unpack the binary package.
APACHE_HADOOP_URL=https://archive.apache.org/dist/hadoop/
HADOOP_VERSION=2.8.5
wget ${APACHE_HADOOP_URL}/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
tar xzvf hadoop-${HADOOP_VERSION}.tar.gz
HADOOP_HOME=`pwd`/hadoop-${HADOOP_VERSION}

export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`

# Start the flink standalone cluster
cd flink-${FLINK_VERSION}/
./bin/start-cluster.sh
```

--------------------------------

### CREATE TABLE with PARTITIONED BY

Source: https://iceberg.apache.org/docs/latest/spark-ddl

Example of creating a table partitioned by a specific column.

```sql
CREATE TABLE prod.db.sample (
    id bigint,
    data string,
    category string)
USING iceberg
PARTITIONED BY (category);
```

--------------------------------

### Download Iceberg Flink dependencies and start SQL client

Source: https://iceberg.apache.org/docs/latest/aws

Download the Iceberg Flink runtime and AWS bundle jars, then start the Flink SQL client shell with those jars. Set ICEBERG_VERSION to the desired version before running.

```bash
# download Iceberg dependency
ICEBERG_VERSION=1.11.0
MAVEN_URL=https://repo1.maven.org/maven2
ICEBERG_MAVEN_URL=$MAVEN_URL/org/apache/iceberg

wget $ICEBERG_MAVEN_URL/iceberg-flink-runtime/$ICEBERG_VERSION/iceberg-flink-runtime-$ICEBERG_VERSION.jar

wget $ICEBERG_MAVEN_URL/iceberg-aws-bundle/$ICEBERG_VERSION/iceberg-aws-bundle-$ICEBERG_VERSION.jar

# start Flink SQL client shell
/path/to/bin/sql-client.sh embedded \
    -j iceberg-flink-runtime-$ICEBERG_VERSION.jar \
    -j iceberg-aws-bundle-$ICEBERG_VERSION.jar \
    shell

```

--------------------------------

### Create table and insert data

Source: https://iceberg.apache.org/docs/latest/branching

Initial setup of a table and insertion of sample data.

```sql
CREATE TABLE db.table (id bigint, data string, col float);
INSERT INTO db.table VALUES (1, 'a', 1.0), (2, 'b', 2.0), (3, 'c', 3.0);
SELECT * FROM db.table;
1   a   1.0
2   b   2.0
3   c   3.0
```

--------------------------------

### Complete example with Hive catalog: create, insert, and query

Source: https://iceberg.apache.org/docs/latest/flink-connector

Complete example using a Hive catalog: creates a Flink table, inserts three rows, sets the result mode to tableau, and queries the table. The output shows the expected result set with three rows.

```sql
CREATE TABLE flink_table (
    id   BIGINT,
    data STRING
) WITH (
    'connector'='iceberg',
    'catalog-name'='hive_prod',
    'uri'='thrift://localhost:9083',
    'warehouse'='file:///path/to/warehouse'
);

INSERT INTO flink_table VALUES (1, 'AAA'), (2, 'BBB'), (3, 'CCC');

SET execution.result-mode=tableau;
SELECT * FROM flink_table;

+----+------+
| id | data |
+----+------+
|  1 |  AAA |
|  2 |  BBB |
|  3 |  CCC |
+----+------+
3 rows in set

```

--------------------------------

### Create destination table for single table example

Source: https://iceberg.apache.org/docs/latest/kafka-connect

Creates the destination table `default.events` for the single destination table example. The table is partitioned by hour on the `ts` column.

```sql
CREATE TABLE default.events (
    id STRING,
    type STRING,
    ts TIMESTAMP,
    payload STRING)
PARTITIONED BY (hours(ts))

```

--------------------------------

### Create and evolve table schema for time travel example

Source: https://iceberg.apache.org/docs/latest/spark-queries

Creates a table with an initial schema, then evolves it by adding a column and inserting new data. The comments record the snapshot IDs and timestamps used in later examples.

```sql
-- snapshot S1: initial schema (id, status)
CREATE TABLE prod.db.orders (
  id BIGINT,
  status STRING
) USING iceberg;

INSERT INTO prod.db.orders VALUES (1, 'NEW'), (2, 'PAID');

-- record snapshot S1's snapshot_id and committed_at timestamp
-- e.g. snapshot_id = 101, committed_at = '2025-01-01 10:00:00'

-- snapshot S2: add a new column "total" and write new data
ALTER TABLE prod.db.orders ADD COLUMN total DOUBLE;

INSERT INTO prod.db.orders VALUES (3, 'PAID', 100.0);

-- now S2 is the current snapshot with schema (id, status, total)

```

--------------------------------

### INSERT INTO with ORDER BY

Source: https://iceberg.apache.org/docs/latest/spark-ddl

Example of an INSERT command using ORDER BY to achieve global ordering.

```sql
INSERT INTO prod.db.sample
SELECT id, data, category, ts FROM another_table
ORDER BY ts, category
```

--------------------------------

### Start Spark SQL shell with Iceberg AWS bundle

Source: https://iceberg.apache.org/docs/latest/aws

Start the Spark SQL shell with the Iceberg Spark runtime and AWS bundle packages, configuring a Glue catalog with S3 warehouse and S3FileIO. Requires Spark 3.4 with Scala 2.12 and the iceberg-aws-bundle for AWS dependencies.

```bash
# start Spark SQL client shell
spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
    --conf spark.sql.defaultCatalog=my_catalog \
    --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \
    --conf spark.sql.catalog.my_catalog.type=glue \
    --conf spark.sql.catalog.my_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO

```

--------------------------------

### Initialize SQL Client with -i option

Source: https://iceberg.apache.org/docs/latest/flink-ddl

Starts the Flink SQL Client with the '-i' option to execute an initialization SQL file, setting up the environment (e.g., catalogs) before the session starts.

```bash
/path/to/bin/sql-client.sh -i /path/to/init.sql
```

--------------------------------

### Start Spark shell with AssumeRoleAwsClientFactory

Source: https://iceberg.apache.org/docs/latest/aws

Start a Spark shell using AssumeRoleAwsClientFactory for cross-account and cross-region access. The factory assumes the specified IAM role (client.assume-role.arn) and region (client.assume-role.region) to initialize Glue, S3, and DynamoDB clients. Requires the iceberg-aws-bundle package and the client.factory property set to org.apache.iceberg.aws.AssumeRoleAwsClientFactory.

```bash
spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
    --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \    
    --conf spark.sql.catalog.my_catalog.type=glue \
    --conf spark.sql.catalog.my_catalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory \
    --conf spark.sql.catalog.my_catalog.client.assume-role.arn=arn:aws:iam::123456789:role/myRoleToAssume \
    --conf spark.sql.catalog.my_catalog.client.assume-role.region=ap-northeast-1

```

--------------------------------

### Setup Flink Environment for Dell ECS

Source: https://iceberg.apache.org/docs/latest/dell

Downloads required dependencies and launches the Flink SQL client with the Iceberg and Dell ECS jars.

```bash
# HADOOP_HOME is your hadoop root directory after unpack the binary package.
export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`

# download Iceberg dependency
MAVEN_URL=https://repo1.maven.org/maven2
ICEBERG_VERSION=0.15.0
FLINK_VERSION=1.14
wget ${MAVEN_URL}/org/apache/iceberg/iceberg-flink-runtime-${FLINK_VERSION}/${ICEBERG_VERSION}/iceberg-flink-runtime-${FLINK_VERSION}-${ICEBERG_VERSION}.jar
wget ${MAVEN_URL}/org/apache/iceberg/iceberg-dell/${ICEBERG_VERSION}/iceberg-dell-${ICEBERG_VERSION}.jar

# download ECS object client
ECS_CLIENT_VERSION=3.3.2
wget ${MAVEN_URL}/com/emc/ecs/object-client-bundle/${ECS_CLIENT_VERSION}/object-client-bundle-${ECS_CLIENT_VERSION}.jar

# open the SQL client.
/path/to/bin/sql-client.sh embedded \
    -j iceberg-flink-runtime-${FLINK_VERSION}-${ICEBERG_VERSION}.jar \
    -j iceberg-dell-${ICEBERG_VERSION}.jar \
    -j object-client-bundle-${ECS_CLIENT_VERSION}.jar \
    shell
```

--------------------------------

### Example file list for table replication

Source: https://iceberg.apache.org/docs/latest/spark-procedures

A sample CSV file format showing the mapping of source paths to target paths for table replication.

```csv
sourcepath/datafile1.parquet,targetpath/datafile1.parquet
sourcepath/datafile2.parquet,targetpath/datafile2.parquet
stagingpath/manifest.avro,targetpath/manifest.avro
```

--------------------------------

### Start Flink SQL Client

Source: https://iceberg.apache.org/docs/latest/flink

Commands to launch the Flink SQL client, noting version-specific differences for loading the Iceberg runtime jar.

```bash
# HADOOP_HOME is your hadoop root directory after unpack the binary package.
export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`   

# Below works for Flink 1.15 or earlier
./bin/sql-client.sh embedded -j <flink-runtime-directory>/iceberg-flink-runtime-2.3-1.11.0.jar shell

# Flink 1.16+ has a regression in loading external jars via -j. See FLINK-30035 for details.
# put iceberg-flink-runtime-2.3-1.11.0.jar in flink/lib dir
./bin/sql-client.sh embedded shell
```

--------------------------------

### CREATE TABLE with hidden partitioning transforms

Source: https://iceberg.apache.org/docs/latest/spark-ddl

Example of creating a table using hidden partition transforms like bucket and days.

```sql
CREATE TABLE prod.db.sample (
    id bigint,
    data string,
    category string,
    ts timestamp)
USING iceberg
PARTITIONED BY (bucket(16, id), days(ts), category);
```

--------------------------------

### UPDATE queries

Source: https://iceberg.apache.org/docs/latest/spark-writes

Examples of UPDATE queries using filters, subqueries, and EXISTS clauses.

```sql
UPDATE prod.db.table
SET c1 = 'update_c1', c2 = 'update_c2'
WHERE ts >= '2020-05-01 00:00:00' and ts < '2020-06-01 00:00:00'

UPDATE prod.db.all_events
SET session_time = 0, ignored = true
WHERE session_time < (SELECT min(session_time) FROM prod.db.good_events)

UPDATE prod.db.orders AS t1
SET order_status = 'returned'
WHERE EXISTS (SELECT oid FROM prod.db.returned_orders WHERE t1.oid = oid)
```

--------------------------------

### Start Spark with MySQL JDBC catalog

Source: https://iceberg.apache.org/docs/latest/jdbc

Start a Spark session with a MySQL JDBC connection using the Iceberg Spark runtime. Configure the catalog type as jdbc, set the JDBC URI, and pass JDBC-specific properties like user, password, and SSL options. Requires the Iceberg Spark runtime package and a MySQL database with atomic transaction support.

```bash
spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0 \
    --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \
    --conf spark.sql.catalog.my_catalog.type=jdbc \
    --conf spark.sql.catalog.my_catalog.uri=jdbc:mysql://test.1234567890.us-west-2.rds.amazonaws.com:3306/default \
    --conf spark.sql.catalog.my_catalog.jdbc.verifyServerCertificate=true \
    --conf spark.sql.catalog.my_catalog.jdbc.useSSL=true \
    --conf spark.sql.catalog.my_catalog.jdbc.user=admin \
    --conf spark.sql.catalog.my_catalog.jdbc.password=pass
```

--------------------------------

### Static overwrite with PARTITION clause

Source: https://iceberg.apache.org/docs/latest/spark-writes

Example query using a PARTITION clause to overwrite specific partitions in static mode.

```sql
INSERT OVERWRITE prod.my_app.logs
PARTITION (level = 'INFO')
SELECT uuid, first(level), first(ts), first(message)
FROM prod.my_app.logs
WHERE level = 'INFO'
GROUP BY uuid
```

--------------------------------

### Start Spark Shell with Iceberg

Source: https://iceberg.apache.org/docs/latest/spark-getting-started

Use the --packages option to include the Iceberg Spark runtime in your Spark shell session.

```bash
spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0
```

--------------------------------

### Get snapshot ancestors by default

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Retrieves all snapshot ancestors for the current state of the specified table.

```sql
CALL spark_catalog.system.ancestors_of('db.tbl');
```

--------------------------------

### Incremental read with start-snapshot-id and end-snapshot-id

Source: https://iceberg.apache.org/docs/latest/spark-queries

Performs an incremental read, getting data appended after the start snapshot (exclusive) up to the end snapshot (inclusive). Only append operations are supported; replace, overwrite, and delete are not. Works with both V1 and V2 format versions, but not supported by Spark SQL syntax.

```scala
// get the data added after start-snapshot-id (10963874102873L) until end-snapshot-id (63874143573109L)
spark.read
  .format("iceberg")
  .option("start-snapshot-id", "10963874102873")
  .option("end-snapshot-id", "63874143573109")
  .load("path/to/table")
```

--------------------------------

### CREATE TABLE LIKE

Source: https://iceberg.apache.org/docs/latest/flink-ddl

Creates a table with the same schema, partitioning, and properties as an existing table. The example shows creating a base table and then a table that copies its definition.

```sql
CREATE TABLE `hive_catalog`.`default`.`sample` (
    id BIGINT COMMENT 'unique id',
    data STRING
);

CREATE TABLE  `hive_catalog`.`default`.`sample_like` LIKE `hive_catalog`.`default`.`sample`;

```

--------------------------------

### Enable prefix listing

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Use prefix-based file listing by setting prefix_listing to true.

```sql
CALL catalog_name.system.remove_orphan_files(table => 'db.sample', prefix_listing => true);
```

--------------------------------

### EMR bootstrap action to install Iceberg dependencies

Source: https://iceberg.apache.org/docs/latest/aws

Bootstrap action for Amazon EMR versions before 6.5.0 to pre-install Apache Iceberg dependencies. Downloads the Iceberg runtime JARs (Spark, Flink, and AWS bundle) from Maven Central into a shared class path. The LIB_PATH is an example shared class path between Spark and Flink; choose a proper class path for production.

```bash
#!/bin/bash

ICEBERG_VERSION=1.11.0
MAVEN_URL=https://repo1.maven.org/maven2
ICEBERG_MAVEN_URL=$MAVEN_URL/org/apache/iceberg
# NOTE: this is just an example shared class path between Spark and Flink,
#  please choose a proper class path for production.
LIB_PATH=/usr/share/aws/aws-java-sdk/


ICEBERG_PACKAGES=(
  "iceberg-spark-runtime-3.5_2.12"
  "iceberg-flink-runtime"
  "iceberg-aws-bundle"
)

install_dependencies () {
  install_path=$1
  download_url=$2
  version=$3
  shift
  pkgs=("$@")
  for pkg in "${pkgs[@]}"; do
    sudo wget -P $install_path $download_url/$pkg/$version/$pkg-$version.jar
  done
}

install_dependencies $LIB_PATH $ICEBERG_MAVEN_URL $ICEBERG_VERSION "${ICEBERG_PACKAGES[@]}"
```

--------------------------------

### Install Apache Flink dependency

Source: https://iceberg.apache.org/docs/latest/flink

Install the required Apache Flink version using pip.

```bash
pip install apache-flink==2.3.0
```

--------------------------------

### cherrypick_snapshot with named args example

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Cherry-picks snapshot 1 using named arguments. Only append and dynamic overwrite snapshots can be cherry-picked. This procedure invalidates all cached Spark plans that reference the affected table.

```sql
CALL catalog_name.system.cherrypick_snapshot(snapshot_id => 1, table => 'my_table' );

```

--------------------------------

### rollback_to_snapshot example

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Rolls back table 'db.sample' to snapshot ID 1. This procedure invalidates all cached Spark plans that reference the affected table.

```sql
CALL catalog_name.system.rollback_to_snapshot('db.sample', 1);

```

--------------------------------

### cherrypick_snapshot example

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Cherry-picks snapshot 1 into the current table state. Only append and dynamic overwrite snapshots can be cherry-picked. This procedure invalidates all cached Spark plans that reference the affected table.

```sql
CALL catalog_name.system.cherrypick_snapshot('my_table', 1);

```

--------------------------------

### Write streaming query to Iceberg table with Hadoop catalog path

Source: https://iceberg.apache.org/docs/latest/spark-structured-streaming

Writes a streaming query to an Iceberg table using the directory-based Hadoop catalog by specifying the table path directly with the "path" option and starting the query with start(). Requires the table to exist before starting the query.

```scala
data.writeStream
    .format("iceberg")
    .outputMode("append")
    .trigger(Trigger.ProcessingTime(1, TimeUnit.MINUTES))
    .option("path", "hdfs://nn:8020/path/to/table") 
    .option("checkpointLocation", checkpointPath)
    .start()

```

--------------------------------

### JsonToMapTransform example JSON input

Source: https://iceberg.apache.org/docs/latest/kafka-connect

Example JSON input for the JsonToMapTransform SMT. It shows a JSON object with primitive, array, empty object, and nested object fields.

```json
{
  "key": 1,
  "array": [1,"two",3],
  "empty_obj": {},
  "nested_obj": {"some_key": ["one", "two"]}
}

```

--------------------------------

### set_current_snapshot by ref example

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Sets the current snapshot for 'db.sample' to tag 's1' using named arguments. Either snapshot_id or ref must be provided but not both. This procedure invalidates all cached Spark plans that reference the affected table.

```sql
CALL catalog_name.system.set_current_snapshot(table => 'db.sample', ref => 's1');

```

--------------------------------

### CALL rewrite_table_path

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Stages a copy of the Iceberg table's metadata files where every absolute path source prefix is replaced by the specified target prefix. This can be the starting point to fully or incrementally copy an Iceberg table to a new location. This procedure only stages rewritten metadata files and prepares a list of files to copy; the actual file copy is not included.

```APIDOC
## CALL rewrite_table_path

### Description
Stages a copy of the Iceberg table's metadata files where every absolute path source prefix is replaced by the specified target prefix. This can be the starting point to fully or incrementally copy an Iceberg table to a new location. This procedure only stages rewritten metadata files and prepares a list of files to copy; the actual file copy is not included.

### Method
CALL

### Endpoint
catalog_name.system.rewrite_table_path

### Parameters
#### Arguments
- **table** (string) - Required - Name of the table
- **source_prefix** (string) - Required - The existing prefix to be replaced
- **target_prefix** (string) - Required - The replacement prefix for source_prefix
- **start_version** (string) - Optional - The name or path of the chronologically first metadata.json to rewrite. Defaults to first metadata.json in table's metadata log.
- **end_version** (string) - Optional - The name or path of the chronologically last metadata.json to rewrite. Defaults to latest metadata.json in table's metadata log.
- **staging_location** (string) - Optional - The output location for newly rewritten metadata files. Defaults to new directory under table's metadata directory.
- **create_file_list** (boolean) - Optional - Whether to generate a file list containing the paths of rewritten metadata. Defaults to true.

### Request Example
```
CALL catalog_name.system.rewrite_table_path(
    table => 'db.my_table',
    source_prefix => 'hdfs://nn:8020/path/to/source_table',
    target_prefix => 's3a://bucket/prefix/db.db/my_table'
);
```

### Response
#### Output
- **latest_version** (string) - Name of the latest metadata file rewritten by this procedure
- **file_list_location** (string) - Path to a CSV file containing a mapping of source to target paths
- **rewritten_manifest_file_paths_count** (int) - Number of manifest files with rewritten paths
- **rewritten_delete_file_paths_count** (int) - Number of delete files with rewritten paths

#### File List
Contains the copy plan for all files added to the table between start_version and end_version. For each file, it specifies:
- Source Path: The original file path in the table, or the staging location if the file has been rewritten
- Target Path: The path with the replacement prefix

Example:
```
sourcepath/datafile1.parquet,targetpath/datafile1.parquet
sourcepath/datafile2.parquet,targetpath/datafile2.parquet
stagingpath/manifest.avro,targetpath/manifest.avro
```
```

--------------------------------

### Create Iceberg Sample Table

Source: https://iceberg.apache.org/docs/latest/spark-writes

SQL command to create a partitioned Iceberg table for demonstration purposes.

```sql
CREATE TABLE prod.db.sample (
    id bigint,
    data string,
    category string,
    ts timestamp)
USING iceberg
PARTITIONED BY (days(ts), category)
```

--------------------------------

### rollback_to_timestamp example

Source: https://iceberg.apache.org/docs/latest/spark-procedures

Rolls back table 'db.sample' to the snapshot current at the specified timestamp. This procedure invalidates all cached Spark plans that reference the affected table.

```sql
CALL catalog_name.system.rollback_to_timestamp('db.sample', TIMESTAMP '2021-06-30 00:00:00.000');

```

--------------------------------

### Time travel metadata tables with SQL

Source: https://iceberg.apache.org/docs/latest/spark-queries

Inspects metadata tables with time travel. The first query gets file manifests at a specific timestamp, and the second gets partitions with a specific snapshot ID.

```sql
-- get the table's file manifests at timestamp Sep 20, 2021 08:00:00
SELECT * FROM prod.db.table.manifests TIMESTAMP AS OF '2021-09-20 08:00:00';

-- get the table's partitions with snapshot id 10963874102873L
SELECT * FROM prod.db.table.partitions VERSION AS OF 10963874102873L;
```

--------------------------------

### GET /tables/{table}/scans/planTasks

Source: https://iceberg.apache.org/docs/latest/api

Returns the tasks for the scan.

```APIDOC
## [GET] /tables/{table}/scans/planTasks

### Description
Returns the tasks for the scan.

### Method
GET

### Endpoint
/tables/{table}/scans/planTasks

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **tasks** (array) - List of tasks.

#### Response Example
{
  "tasks": []
}

```

--------------------------------

### Query all manifest files

Source: https://iceberg.apache.org/docs/latest/flink-queries

Use the all_manifests metadata table to list all manifest files associated with the table across all snapshots.

```sql
SELECT * FROM prod.db.table$all_manifests;
```

--------------------------------

### Initialize a HadoopCatalog

Source: https://iceberg.apache.org/docs/latest/java-api-quickstart

Creates a HadoopCatalog instance using a Hadoop Configuration and a warehouse path. This catalog requires a file system that supports atomic rename.

```java
import org.apache.hadoop.conf.Configuration;
import org.apache.iceberg.hadoop.HadoopCatalog;

Configuration conf = new Configuration();
String warehousePath = "hdfs://host:8020/warehouse_path";
HadoopCatalog catalog = new HadoopCatalog(conf, warehousePath);
```

--------------------------------

### Create catalog through SQL files

Source: https://iceberg.apache.org/docs/latest/flink-ddl

Defines an initialization SQL file that creates a Hive catalog and sets it as the current catalog. This file is used with the '-i' startup option to set up the SQL Client environment.

```sql
-- define available catalogs
CREATE CATALOG hive_catalog WITH (
  'type'='iceberg',
  'catalog-type'='hive',
  'uri'='thrift://localhost:9083',
  'warehouse'='hdfs://nn:8020/warehouse/path'
);

USE CATALOG hive_catalog;
```

--------------------------------

### GET /tables/{table}/scans/planFiles

Source: https://iceberg.apache.org/docs/latest/api

Returns the files that match the scan.

```APIDOC
## [GET] /tables/{table}/scans/planFiles

### Description
Returns the files that match the scan.

### Method
GET

### Endpoint
/tables/{table}/scans/planFiles

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **files** (array) - List of files.

#### Response Example
{
  "files": []
}

```

--------------------------------

### GET /tables/{table}/snapshots

Source: https://iceberg.apache.org/docs/latest/api

Returns all valid snapshots for the table.

```APIDOC
## [GET] /tables/{table}/snapshots

### Description
Returns all valid snapshots for the table.

### Method
GET

### Endpoint
/tables/{table}/snapshots

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **snapshots** (array) - List of valid snapshots.

#### Response Example
{
  "snapshots": []
}

```

--------------------------------

### GET /tables/{table}/snapshots/current

Source: https://iceberg.apache.org/docs/latest/api

Returns the current table snapshot.

```APIDOC
## [GET] /tables/{table}/snapshots/current

### Description
Returns the current table snapshot.

### Method
GET

### Endpoint
/tables/{table}/snapshots/current

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **snapshot** (object) - The current table snapshot.

#### Response Example
{
  "snapshot": {}
}

```

--------------------------------

### Load Nessie catalog in Java

Source: https://iceberg.apache.org/docs/latest/nessie

Load a Nessie catalog directly in Java using CatalogUtil. Requires the warehouse path, Nessie server URI, and optional ref (branch or tag).

```java
Map<String, String> options = new HashMap<>();
options.put("warehouse", "/path/to/warehouse");
options.put("ref", "main");
options.put("uri", "https://localhost:19120/api/v2");
Catalog nessieCatalog = CatalogUtil.loadCatalog("org.apache.iceberg.nessie.NessieCatalog", "nessie", options, hadoopConfig);

```

--------------------------------

### GET /tables/{table}/schema

Source: https://iceberg.apache.org/docs/latest/api

Returns the current table schema.

```APIDOC
## [GET] /tables/{table}/schema

### Description
Returns the current table schema.

### Method
GET

### Endpoint
/tables/{table}/schema

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **schema** (object) - The current table schema.

#### Response Example
{
  "schema": {}
}

```

--------------------------------

### Show views

Source: https://iceberg.apache.org/docs/latest/spark-ddl

Lists views in the current namespace or specified catalog/namespace.

```sql
SHOW VIEWS
```

```sql
SHOW VIEWS IN <catalog>
```

```sql
SHOW VIEWS IN <namespace>
```

```sql
SHOW VIEWS IN <catalog>.<namespace>
```

--------------------------------

### GET /tables/{table}/scans/schema

Source: https://iceberg.apache.org/docs/latest/api

Returns the read projection schema for the scan.

```APIDOC
## [GET] /tables/{table}/scans/schema

### Description
Returns the read projection schema for the scan.

### Method
GET

### Endpoint
/tables/{table}/scans/schema

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **schema** (object) - The read projection schema.

#### Response Example
{
  "schema": {}
}

```

--------------------------------

### GET /tables/{table}/location

Source: https://iceberg.apache.org/docs/latest/api

Returns the table's base location.

```APIDOC
## [GET] /tables/{table}/location

### Description
Returns the table's base location.

### Method
GET

### Endpoint
/tables/{table}/location

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **location** (string) - The table's base location.

#### Response Example
{
  "location": ""
}

```

--------------------------------

### Create and query a branch

Source: https://iceberg.apache.org/docs/latest/branching

Create a branch named test_branch pointing to the current snapshot and read data from it.

```sql
ALTER TABLE db.table CREATE BRANCH test_branch;

SELECT * FROM db.table.branch_test_branch;
1   a   1.0
2   b   2.0
3   c   3.0
```

--------------------------------

### GET /tables/{table}/snapshots/{id}

Source: https://iceberg.apache.org/docs/latest/api

Returns a specific snapshot by ID.

```APIDOC
## [GET] /tables/{table}/snapshots/{id}

### Description
Returns a specific snapshot by ID.

### Method
GET

### Endpoint
/tables/{table}/snapshots/{id}

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.
- **id** (long) - Required - The snapshot ID.

### Response
#### Success Response (200)
- **snapshot** (object) - The requested snapshot.

#### Response Example
{
  "snapshot": {}
}

```

--------------------------------

### Show CREATE statement

Source: https://iceberg.apache.org/docs/latest/spark-ddl

Displays the SQL statement used to create the specified view.

```sql
SHOW CREATE TABLE <viewName>
```

--------------------------------

### GET /tables/{table}/properties

Source: https://iceberg.apache.org/docs/latest/api

Returns a map of key-value properties for the table.

```APIDOC
## [GET] /tables/{table}/properties

### Description
Returns a map of key-value properties for the table.

### Method
GET

### Endpoint
/tables/{table}/properties

### Parameters
#### Path Parameters
- **table** (string) - Required - The table identifier.

### Response
#### Success Response (200)
- **properties** (map) - Key-value properties.

#### Response Example
{
  "properties": {}
}

```
Tokens:
Raw