PyIceberg

https://py.iceberg.apache.org/
Admin
PyIceberg is a Python implementation for accessing Apache Iceberg tables without requiring a JVM,...
Tokens:438,075
Snippets:3,549
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Launch PyIceberg Examples Notebook

Source: https://py.iceberg.apache.org/contributing

Installs notebook dependencies and launches Jupyter Lab in the notebooks directory for basic PyIceberg experimentation.

```bash
make notebook

```

--------------------------------

### Install uv and Set Up Development Environment

Source: https://py.iceberg.apache.org/contributing

Installs uv if needed, creates a virtual environment, and installs all project dependencies. This is the primary command for setting up a new development environment.

```bash
make install

```

--------------------------------

### Download Sample Data

Source: https://py.iceberg.apache.org/

Downloads a sample Parquet file for use in subsequent examples. Ensure curl is installed and accessible.

```bash
curl https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-01.parquet -o /tmp/yellow_tripdata_2023-01.parquet

```

--------------------------------

### REST Catalog Configuration

Source: https://py.iceberg.apache.org/configuration

Configuration examples for connecting to a REST catalog, including basic and mTLS secured setups.

```yaml
catalog:
  default:
    uri: http://rest-catalog/ws/
    credential: t-1234:secret

  default-mtls-secured-catalog:
    uri: https://rest-catalog/ws/
    ssl:
      client:
        cert: /absolute/path/to/client.crt
        key: /absolute/path/to/client.key
      cabundle: /absolute/path/to/cabundle.pem
```

--------------------------------

### Install PyIceberg Directly from GitHub

Source: https://py.iceberg.apache.org/contributing

Installs PyIceberg directly from its GitHub repository. This is generally not recommended for production but can be useful for testing specific branches or forks.

```bash
pip install "git+https://github.com/apache/iceberg-python.git#egg=pyiceberg[pyarrow]"

```

--------------------------------

### Install uv Only

Source: https://py.iceberg.apache.org/contributing

Installs only the uv tool without setting up the project's virtual environment or dependencies. Use this if you only need uv for other purposes.

```bash
make install-uv

```

--------------------------------

### Install Polars Integration for PyIceberg

Source: https://py.iceberg.apache.org/api

Installs PyIceberg with the necessary dependencies for Polars integration.

```bash
pip install pyiceberg['polars']
```

--------------------------------

### Install Bodo with PyIceberg

Source: https://py.iceberg.apache.org/api

Install the PyIceberg library with Bodo support using pip. This is a prerequisite for using Bodo DataFrames with Iceberg tables.

```bash
pip install pyiceberg['bodo']
```

--------------------------------

### Install PyIceberg with S3FS and Hive support

Source: https://py.iceberg.apache.org/

Install the latest release version of PyIceberg from PyPI, including optional dependencies for S3FS and Hive.

```bash
pip install "pyiceberg[s3fs,hive]"

```

--------------------------------

### Install PyIceberg Nightly Build

Source: https://py.iceberg.apache.org/nightly-build

Use this command to install the latest nightly build of PyIceberg from TestPyPI. This build is intended for testing purposes only.

```bash
pip install -i https://test.pypi.org/simple/ --pre pyiceberg

```

--------------------------------

### Install PyIceberg from Source with S3FS and Hive Support

Source: https://py.iceberg.apache.org/contributing

Installs PyIceberg in editable mode from a local clone of the repository, including optional dependencies for S3 and Hive support.

```bash
git clone https://github.com/apache/iceberg-python.git
cd iceberg-python
pip3 install -e ".[s3fs,hive]"

```

--------------------------------

### Partition Exclusion Example Path

Source: https://py.iceberg.apache.org/configuration

Shows a data file path with partition keys and values omitted for further optimization.

```text
s3://bucket/ns/table/data/1101/0100/1011/00111010-00000-0-5affc076-96a4-48f2-9cd2-d5efbc9f0c94-00001.parquet
```

--------------------------------

### PyArrow Table Schema and Data Example

Source: https://py.iceberg.apache.org/api

This example shows the structure of a PyArrow Table returned by `table.inspect.entries()`. It includes the schema definition and a sample data row, illustrating the format of manifest entries.

```python
pyarrow.Table
status: int8 not null
snapshot_id: int64 not null
sequence_number: int64 not null
file_sequence_number: int64 not null
data_file: struct<content: int8 not null, file_path: string not null, file_format: string not null, partition: struct<> not null, record_count: int64 not null, file_size_in_bytes: int64 not null, column_sizes: map<int32, int64>, value_counts: map<int32, int64>, null_value_counts: map<int32, int64>, nan_value_counts: map<int32, int64>, lower_bounds: map<int32, binary>, upper_bounds: map<int32, binary>, key_metadata: binary, split_offsets: list<item: int64>, equality_ids: list<item: int32>, sort_order_id: int32> not null
  child 0, content: int8 not null
  child 1, file_path: string not null
  child 2, file_format: string not null
  child 3, partition: struct<> not null
  child 4, record_count: int64 not null
  child 5, file_size_in_bytes: int64 not null
  child 6, column_sizes: map<int32, int64>
      child 0, entries: struct<key: int32 not null, value: int64> not null
          child 0, key: int32 not null
          child 1, value: int64
  child 7, value_counts: map<int32, int64>
      child 0, entries: struct<key: int32 not null, value: int64> not null
          child 0, key: int32 not null
          child 1, value: int64
  child 8, null_value_counts: map<int32, int64>
      child 0, entries: struct<key: int32 not null, value: int64> not null
          child 0, key: int32 not null
          child 1, value: int64
  child 9, nan_value_counts: map<int32, int64>
      child 0, entries: struct<key: int32 not null, value: int64> not null
          child 0, key: int32 not null
          child 1, value: int64
  child 10, lower_bounds: map<int32, binary>
      child 0, entries: struct<key: int32 not null, value: binary> not null
          child 0, key: int32 not null
          child 1, value: binary
  child 11, upper_bounds: map<int32, binary>
      child 0, entries: struct<key: int32 not null, value: binary> not null
          child 0, key: int32 not null
          child 1, value: binary
  child 12, key_metadata: binary
  child 13, split_offsets: list<item: int64>
      child 0, item: int64
  child 14, equality_ids: list<item: int32>
      child 0, item: int32
  child 15, sort_order_id: int32
readable_metrics: struct<city: struct<column_size: int64, value_count: int64, null_value_count: int64, nan_value_count: int64, lower_bound: string, upper_bound: string> not null, lat: struct<column_size: int64, value_count: int64, null_value_count: int64, nan_value_count: int64, lower_bound: double, upper_bound: double> not null, long: struct<column_size: int64, value_count: int64, null_value_count: int64, nan_value_count: int64, lower_bound: double, upper_bound: double> not null>
  child 0, city: struct<column_size: int64, value_count: int64, null_value_count: int64, nan_value_count: int64, lower_bound: string, upper_bound: string> not null
      child 0, column_size: int64
      child 1, value_count: int64
      child 2, null_value_count: int64
      child 3, nan_value_count: int64
      child 4, lower_bound: string
      child 5, upper_bound: string
  child 1, lat: struct<column_size: int64, value_count: int64, null_value_count: int64, nan_value_count: int64, lower_bound: double, upper_bound: double> not null
      child 0, column_size: int64
      child 1, value_count: int64
      child 2, null_value_count: int64
      child 3, nan_value_count: int64
      child 4, lower_bound: double
      child 5, upper_bound: double
  child 2, long: struct<column_size: int64, value_count: int64, null_value_count: int64, nan_value_count: int64, lower_bound: double, upper_bound: double> not null
      child 0, column_size: int64
      child 1, value_count: int64
      child 2, null_value_count: int64
      child 3, nan_value_count: int64
      child 4, lower_bound: double
      child 5, upper_bound: double
----
status: [[1]]
snapshot_id: [[6245626162224016531]]
sequence_number: [[1]]
file_sequence_number: [[1]]
data_file: [
  -- is_valid: all not null
  -- child 0 type: int8
[0]
  -- child 1 type: string
["s3://warehouse/default/cities/data/00000-0-80766b66-e558-4150-a5cf-85e4c609b9fe.parquet"]
  -- child 2 type: string
["PARQUET"]
  -- child 3 type: struct<>
    -- is_valid: all not null
  -- child 4 type: int64
[4]
  -- child 5 type: int64
[1656]
  -- child 6 type: map<int32, int64>
[keys:[1,2,3]values:[140,135,135]]
  -- child 7 type: map<int32, int64>
[keys:[1,2,3]values:[4,4,4]]
  -- child 8 type: map<int32, int64>
[keys:[1,2,3]values:[0,0,0]]
  -- child 9 type: map<int32, int64>
[keys:[]values:[]]
  -- child 10 type: map<int32, binary>
[keys:[1,2,3]values:[416D7374657264616D,8602B68311E34240,3A77BB5E9A9B5EC0]]
  -- child 11 type: map<int32, binary>
[keys:[1,2,3]values:[53616E204672616E636973636F,F5BEF1B5678E4A40,304CA60A46651840]]
  -- child 12 type: binary
[null]

```

--------------------------------

### Basic Filtering Examples

Source: https://py.iceberg.apache.org/expression-dsl

Demonstrates creating filters for active adult users, high-value customers, and date ranges.

```python
from datetime import datetime
from pyiceberg.expressions import (
    And,
    EqualTo,
    GreaterThanOrEqual,
    LessThanOrEqual,
    GreaterThan,
    In
)

active_adult_users_filter = And(
    EqualTo("status", "active"),
    GreaterThanOrEqual("age", 18)
)


high_value_customers = And(
    GreaterThan("total_spent", 1000),
    In("membership_level", ["gold", "platinum"])
)

date_range_filter = And(
    GreaterThanOrEqual("created_at", datetime(2024, 1, 1)),
    LessThanOrEqual("created_at", datetime(2024, 12, 31))
)
```

--------------------------------

### Print PyIceberg Version

Source: https://py.iceberg.apache.org/reference/pyiceberg/cli/console

Prints the currently installed PyIceberg version to the console.

```python
import click
from pyiceberg.cli.console import run, catch_exception
from pyiceberg.version import __version__

@run.command()
@click.pass_context
@catch_exception()
def version(ctx: Context) -> None:
    """Print pyiceberg version."""
    ctx.obj["output"].version(__version__)
```

--------------------------------

### Configure REST Catalog in YAML

Source: https://py.iceberg.apache.org/contributing

Example configuration for a REST catalog in the ~/.pyiceberg.yaml file, specifying the URI and credentials.

```yaml
catalog:
  test_catalog:
    uri: http://rest-catalog/ws/
    credential: t-1234:secret

```

--------------------------------

### PyArrow Table Schema Example

Source: https://py.iceberg.apache.org/api

This is an example representation of a PyArrow table schema, showing column names and data types.

```python
pyarrow.Table
VendorID: int64
tpep_pickup_datetime: timestamp[us, tz=+00:00]
tpep_dropoff_datetime: timestamp[us, tz=+00:00]
----
VendorID: [[2,1,2,1,1,...,2,2,2,2,2],[2,1,1,1,2,...,1,1,2,1,2],...,[2,2,2,2,2,...,2,6,6,2,2],[2,2,2,2,2,...,2,2,2,2,2]]
tpep_pickup_datetime: [[2021-04-01 00:28:05.000000,...,2021-04-30 23:44:25.000000]]
tpep_dropoff_datetime: [[2021-04-01 00:47:59.000000,...,2021-05-01 00:14:47.000000]]


```

--------------------------------

### SimpleLocationProvider Data File Path (Non-Partitioned)

Source: https://py.iceberg.apache.org/configuration

Example of a data file location managed by SimpleLocationProvider for a non-partitioned table.

```text
s3://bucket/ns/table/data/0000-0-5affc076-96a4-48f2-9cd2-d5efbc9f0c94-00001.parquet
```

--------------------------------

### Example Identifier Tuple

Source: https://py.iceberg.apache.org/reference/pyiceberg/typedef

Demonstrates how to create an Identifier, which is a tuple of strings representing a table's unique path.

```python
("namespace", "table_name")
```

--------------------------------

### Create Namespace

Source: https://py.iceberg.apache.org/api

Creates a new namespace named 'docs_example' within the loaded catalog.

```python
catalog.create_namespace("docs_example")

```

--------------------------------

### BoundStartsWith Class Definition

Source: https://py.iceberg.apache.org/reference/pyiceberg/expressions

Defines a bound predicate for 'starts with' operations. It inherits from BoundLiteralPredicate and provides a method to get its unbound equivalent.

```python
class BoundStartsWith(BoundLiteralPredicate):
    def __invert__(self) -> BoundNotStartsWith:
        """Transform the Expression into its negated version."""
        return BoundNotStartsWith(self.term, self.literal)

    @property
    def as_unbound(self) -> type[StartsWith]:
        return StartsWith

```

--------------------------------

### Pyiceberg CLI Help

Source: https://py.iceberg.apache.org/cli

Displays the available options and commands for the Pyiceberg CLI. Use this to understand the CLI's capabilities.

```bash
➜  pyiceberg --help
Usage: pyiceberg [OPTIONS] COMMAND [ARGS]...

Options:
  --catalog TEXT
  --verbose BOOLEAN
  --output [text|json]
  --ugi TEXT
  --uri TEXT
  --credential TEXT
  --help                Show this message and exit.

Commands:
  create      Operation to create a namespace.
  describe    Describe a namespace or a table.
  drop        Operations to drop a namespace or table.
  files       List all the files of the table.
  list        List tables or namespaces.
  list-refs   List all the refs in the provided table.
  location    Return the location of the table.
  properties  Properties on tables/namespaces.
  rename      Rename a table.
  schema      Get the schema of the table.
  spec        Return the partition spec of the table.
  uuid        Return the UUID of the table.
  version     Print pyiceberg version.


```

--------------------------------

### Object Store Location Provider Example Path

Source: https://py.iceberg.apache.org/configuration

Illustrates a data file path with injected binary directories for object store prefix distribution.

```text
s3://bucket/ns/table/data/0101/0110/1001/10110010/category=orders/0000-0-5affc076-96a4-48f2-9cd2-d5efbc9f0c94-00001.parquet
```

--------------------------------

### Get Default Warehouse Location (Hive Style)

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/glue

Overrides the default warehouse location to follow Hive-style conventions. This is useful for maintaining compatibility with existing Hive setups.

```python
def _get_default_warehouse_location(self, database_name: str, table_name: str) -> str:
        """Override the default warehouse location to follow Hive-style conventions."""
        return self._get_hive_style_warehouse_location(database_name, table_name)
```

--------------------------------

### ancestors_of

Source: https://py.iceberg.apache.org/reference/pyiceberg/table/snapshots

Get the ancestors of and including the given snapshot. This function traverses the parent snapshot IDs starting from the `current_snapshot` until it reaches a snapshot with no parent, yielding each snapshot along the way.

```APIDOC
## ancestors_of(current_snapshot, table_metadata)

### Description
Get the ancestors of and including the given snapshot.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **snapshot** (Snapshot) - An ancestor snapshot.

#### Response Example
None
```

--------------------------------

### Initialize GlueCatalog with a GlueClient

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/glue

Instantiate GlueCatalog by providing an existing boto3 GlueClient. This is useful when you have already configured and authenticated a client.

```python
from pyiceberg.catalog.glue import GlueCatalog
from botocore.client import GlueClient

# Assume 'glue_client' is an initialized boto3 GlueClient
glu_catalog = GlueCatalog(name="my_glue_catalog", client=glue_client)
```

--------------------------------

### List Namespaces

Source: https://py.iceberg.apache.org/api

Lists all existing namespaces in the catalog and asserts that 'docs_example' is present.

```python
ns = catalog.list_namespaces()

assert ns == [("docs_example",)]

```

--------------------------------

### Example Identifier Assignment

Source: https://py.iceberg.apache.org/reference/pyiceberg/typedef

Shows an example of assigning a tuple to the Identifier type.

```python
>>> identifier: Identifier = ("namespace", "table_name")
```

--------------------------------

### __init__

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/dynamodb

Initializes the DynamoDB Catalog. It can optionally take a pre-configured boto3 DynamoDB client or construct one using provided properties.

```APIDOC
## __init__ (name, client=None, **properties)

### Description
Initializes the DynamoDB Catalog. It can optionally take a pre-configured boto3 DynamoDB client or construct one using provided properties.

### Parameters
#### Path Parameters
- **name** (str) - Required - Name to identify the catalog.
- **client** (Optional[DynamoDBClient]) - Optional - An optional boto3 dynamodb client.
- **properties** (str) - Optional - Properties for dynamodb client construction and configuration.
```

--------------------------------

### MetastoreCatalog Initialization

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog

Initializes a MetastoreCatalog instance with a name and properties.

```python
class MetastoreCatalog(Catalog, ABC):
    def __init__(self, name: str, **properties: str):
        super().__init__(name, **properties)

```

--------------------------------

### Initialize BigQueryMetastoreCatalog

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/bigquery_metastore

Initializes the BigQueryMetastoreCatalog with necessary GCP configurations. Requires GCP project ID and enables 'legacy-current-snapshot-id'. Supports credentials via file path or JSON info.

```python
class BigQueryMetastoreCatalog(MetastoreCatalog):
    def __init__(self, name: str, **properties: str):
        super().__init__(name, **properties)

        project_id: str | None = self.properties.get(GCP_PROJECT_ID)
        location: str | None = self.properties.get(GCP_LOCATION)
        credentials_file: str | None = self.properties.get(GCP_CREDENTIALS_FILE)
        credentials_info_str: str | None = self.properties.get(GCP_CREDENTIALS_INFO)

        if not project_id:
            raise ValueError(f"Missing property: {GCP_PROJECT_ID}")

        # BigQuery requires current-snapshot-id to be present for tables to be created.
        if not Config().get_bool("legacy-current-snapshot-id"):
            raise ValueError("legacy-current-snapshot-id must be enabled to work with BigQuery.")

        if credentials_file and credentials_info_str:
            raise ValueError("Cannot specify both `gcp.bigquery.credentials-file` and `gcp.bigquery.credentials-info`")

        gcp_credentials = None
        if credentials_file:
            gcp_credentials = service_account.Credentials.from_service_account_file(credentials_file)
        elif credentials_info_str:
            try:
                credentials_info_dict = json.loads(credentials_info_str)
                gcp_credentials = service_account.Credentials.from_service_account_info(credentials_info_dict)
            except json.JSONDecodeError as e:
                raise ValueError(f"Invalid JSON string for {GCP_CREDENTIALS_INFO}: {e}") from e
            except TypeError as e:  # from_service_account_info can raise TypeError for bad structure
                raise ValueError(f"Invalid credentials structure for {GCP_CREDENTIALS_INFO}: {e}") from e

        self.client: Client = Client(
            project=project_id,
            credentials=gcp_credentials,
            location=location,
        )

        self.location = location
        self.project_id = project_id
```

--------------------------------

### Deprecation Message Example

Source: https://py.iceberg.apache.org/how-to-release

Example of a deprecation message function. The behavior needs to be adjusted based on the message content.

```python
deprecation_message(
    deprecated_in="0.1.0",
    removed_in="0.2.0",
    help_message="The old_property is deprecated. Please use the something_else property instead.",
)

```

--------------------------------

### LongType Class Definition and Example

Source: https://py.iceberg.apache.org/reference/pyiceberg/types

Defines the LongType for 64-bit signed integers in Iceberg and provides an example of its usage and attributes.

```python
class LongType(PrimitiveType):
    """A Long data type in Iceberg can be represented using an instance of this class.

    Longs in Iceberg are 64-bit signed integers.

    Example:
        >>> column_foo = LongType()
        >>> isinstance(column_foo, LongType)
        True
        >>> column_foo
        LongType()
        >>> str(column_foo)
        'long'

    Attributes:
        max (int): The maximum allowed value for Longs, inherited from the canonical Iceberg implementation
            in Java. (returns `9223372036854775807`)
        min (int): The minimum allowed value for Longs, inherited from the canonical Iceberg implementation
            in Java (returns `-9223372036854775808`)
    """

    root: Literal["long"] = Field(default="long")

    max: ClassVar[int] = 9223372036854775807
    min: ClassVar[int] = -9223372036854775808
```

--------------------------------

### Loading Configuration from Files

Source: https://py.iceberg.apache.org/reference/pyiceberg/utils/config

Loads configuration from YAML files. It searches for 'pyiceberg.yml' in PYICEBERG_HOME, the user's home directory, and the current directory.

```python
@staticmethod
def _from_configuration_files() -> RecursiveDict | None:
    """Load the first configuration file that its finds.

    Will first look in the PYICEBERG_HOME env variable,
    and then in the home directory.
    """

    def _load_yaml(directory: str | None) -> RecursiveDict | None:
        if directory:
            path = os.path.join(directory, PYICEBERG_YML)
            if os.path.isfile(path):
                with open(path, encoding=UTF8) as f:
                    yml_str = f.read()
                file_config = strictyaml.load(yml_str).data
                file_config_lowercase = _lowercase_dictionary_keys(file_config)
                return file_config_lowercase
        return None

    # Directories to search for the configuration file
    # The current search order is: PYICEBERG_HOME, home directory, then current directory
    search_dirs = [os.environ.get(PYICEBERG_HOME), os.path.expanduser("~"), os.getcwd()]

    for directory in search_dirs:
        if config := _load_yaml(directory):
            return config

    # Didn't find a config
    return None
```

--------------------------------

### Launch Spark Integration Notebook Environment

Source: https://py.iceberg.apache.org/contributing

Spins up the full integration test infrastructure via Docker Compose, including Spark, REST Catalog, Hive Metastore, and S3-compatible storage.

```bash
make notebook-infra

```

--------------------------------

### Initialize Glue Catalog with Client or Properties

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/glue

Initializes the Glue Catalog. You can either provide an existing boto3 Glue client or let the catalog construct one using provided properties. Properties can configure session details, endpoint URLs, and retry configurations.

```python
def __init__(self, name: str, client: Optional["GlueClient"] = None, **properties: Any):
    """Glue Catalog.

    You either need to provide a boto3 glue client, or one will be constructed from the properties.

    Args:
        name: Name to identify the catalog.
        client: An optional boto3 glue client.
        properties: Properties for glue client construction and configuration.
    """
    super().__init__(name, **properties)

    if client is not None:
        self.glue = client
    else:
        retry_mode_prop_value = get_first_property_value(properties, GLUE_RETRY_MODE)

        session = boto3.Session(
            profile_name=get_first_property_value(properties, GLUE_PROFILE_NAME, AWS_PROFILE_NAME),
            region_name=get_first_property_value(properties, GLUE_REGION, AWS_REGION),
            botocore_session=properties.get(BOTOCORE_SESSION),
            aws_access_key_id=get_first_property_value(properties, GLUE_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
            aws_secret_access_key=get_first_property_value(properties, GLUE_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
            aws_session_token=get_first_property_value(properties, GLUE_SESSION_TOKEN, AWS_SESSION_TOKEN),
        )
        self.glue: GlueClient = session.client(
            "glue",
            endpoint_url=properties.get(GLUE_CATALOG_ENDPOINT),
            config=Config(
                retries={
                    "max_attempts": properties.get(GLUE_MAX_RETRIES, MAX_RETRIES),
                    "mode": retry_mode_prop_value if retry_mode_prop_value in EXISTING_RETRY_MODES else STANDARD_RETRY_MODE,
                }
            ),
        )

        if glue_catalog_id := properties.get(GLUE_ID):
            _register_glue_catalog_id_with_glue_client(self.glue, glue_catalog_id)
```

--------------------------------

### Evaluate Not Starts With Bound Term

Source: https://py.iceberg.apache.org/reference/pyiceberg/expressions/visitors

Evaluates if a bound term (string) does not start with a given literal value. Returns AlwaysTrue or AlwaysFalse.

```python
def visit_not_starts_with(self, term: BoundTerm, literal: LiteralValue) -> BooleanExpression:
    if not self.visit_starts_with(term, literal):
        return AlwaysTrue()
    else:
        return AlwaysFalse()
```

--------------------------------

### Create Table with PyIceberg Schema

Source: https://py.iceberg.apache.org/api

Creates a table named 'docs_example.bids' with a defined PyIceberg schema, partition specification, and sort order.

```python
from pyiceberg.schema import Schema
from pyiceberg.types import (
    TimestampType,
    FloatType,
    DoubleType,
    StringType,
    NestedField,
    StructType,
)

schema = Schema(
    NestedField(field_id=1, name="datetime", field_type=TimestampType(), required=True),
    NestedField(field_id=2, name="symbol", field_type=StringType(), required=True),
    NestedField(field_id=3, name="bid", field_type=FloatType(), required=False),
    NestedField(field_id=4, name="ask", field_type=DoubleType(), required=False),
    NestedField(
        field_id=5,
        name="details",
        field_type=StructType(
            NestedField(
                field_id=4, name="created_by", field_type=StringType(), required=False
            ),
        ),
        required=False,
    ),
)

from pyiceberg.partitioning import PartitionSpec, PartitionField

partition_spec = PartitionSpec(
    PartitionField(
        source_id=1, field_id=1000, transform="day", name="datetime_day"
    )
)

from pyiceberg.table.sorting import SortOrder, SortField

# Sort on the symbol
sort_order = SortOrder(SortField(source_id=2, transform='identity'))

catalog.create_table(
    identifier="docs_example.bids",
    schema=schema,
    partition_spec=partition_spec,
    sort_order=sort_order,
)

```

--------------------------------

### Import GPG Keys

Source: https://py.iceberg.apache.org/verify-release

Download the Apache Iceberg KEYS file and import it into your GPG keyring. This is the first step in verifying release signatures.

```bash
curl https://downloads.apache.org/iceberg/KEYS -o KEYS
gpg --import KEYS

```

--------------------------------

### StringType Class Definition and Example

Source: https://py.iceberg.apache.org/reference/pyiceberg/types

Defines the StringType for arbitrary-length UTF-8 encoded character sequences in Iceberg. Includes an example of instantiation and type checking.

```python
class StringType(PrimitiveType):
    """A String data type in Iceberg can be represented using an instance of this class.

    Strings in Iceberg are arbitrary-length character sequences and are encoded with UTF-8.

    Example:
        >>> column_foo = StringType()
        >>> isinstance(column_foo, StringType)
        True
        >>> column_foo
        StringType()
    """

    root: Literal["string"] = Field(default="string")
```

--------------------------------

### Initialize GlueCatalog with properties

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/glue

Instantiate GlueCatalog by providing properties, which will be used to construct a boto3 GlueClient. This allows for configuration of AWS credentials, region, and endpoint.

```python
from pyiceberg.catalog.glue import GlueCatalog

glu_catalog = GlueCatalog(
    name="my_glue_catalog",
    properties={
        "aws_access_key_id": "YOUR_ACCESS_KEY",
        "aws_secret_access_key": "YOUR_SECRET_KEY",
        "aws_region": "us-east-1",
        "glue_catalog_id": "YOUR_GLUE_CATALOG_ID",
        "glue_endpoint": "https://glue.amazonaws.com"
    }
)
```

--------------------------------

### Remove Deprecated API Example

Source: https://py.iceberg.apache.org/how-to-release

Example of a deprecated API that should be removed before preparing a release. Ensure the help message is updated according to the deprecation notes.

```python
@deprecated(
    deprecated_in="0.1.0",
    removed_in="0.2.0",
    help_message="Please use load_something_else() instead",
)

```

--------------------------------

### __enter__

Source: https://py.iceberg.apache.org/reference/pyiceberg/manifest

Opens the manifest writer, preparing it for adding entries. This method is typically used in a context management scenario.

```APIDOC
## __enter__()

### Description
Opens the manifest writer. This method is intended to be used with Python's `with` statement for automatic resource management.

### Method
`__enter__`

### Parameters
None

### Returns
- **ManifestWriter**: The opened manifest writer instance.
```

--------------------------------

### Evaluate Starts With Bound Term

Source: https://py.iceberg.apache.org/reference/pyiceberg/expressions/visitors

Evaluates if a bound term (string) starts with a given literal value. Handles None values and returns AlwaysTrue or AlwaysFalse.

```python
def visit_starts_with(self, term: BoundTerm, literal: LiteralValue) -> BooleanExpression:
    eval_res = term.eval(self.struct)
    if eval_res is not None and str(eval_res).startswith(str(literal.value)):
        return AlwaysTrue()
    else:
        return AlwaysFalse()
```

--------------------------------

### __enter__

Source: https://py.iceberg.apache.org/reference/pyiceberg/manifest

Opens the ManifestListWriter for writing operations.

```APIDOC
## __enter__()

### Description
Opens the ManifestListWriter, preparing it to write manifest entries. This method is typically used in a `with` statement.

### Returns
- **ManifestListWriter** - The opened writer instance.
```

--------------------------------

### AvroFile __enter__ Method

Source: https://py.iceberg.apache.org/reference/pyiceberg/avro/file

Enters a context manager for AvroFile, initializing the decoder, header, schema, and reader. Use this to prepare an AvroFile for iteration.

```python
def __enter__(self) -> AvroFile[D]:
    """Generate a reader tree for the payload within an avro file.

    Return:
        A generator returning the AvroStructs.
    """
    with self.input_file.open() as f:
        self.decoder = new_decoder(f.read())
    self.header = self._read_header()
    self.schema = self.header.get_schema()
    if not self.read_schema:
        self.read_schema = self.schema

    self.reader = resolve_reader(self.schema, self.read_schema, self.read_types, self.read_enums)

    return self
```

--------------------------------

### EntraAuthManager Class Definition

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/rest/auth

Defines the EntraAuthManager for Azure AD authentication. It uses DefaultAzureCredential to handle multiple authentication methods automatically. Install with `pip install pyiceberg[entra-auth]`.

```python
class EntraAuthManager(AuthManager):
    """Auth Manager implementation that supports Microsoft Entra ID (Azure AD) authentication.

    This manager uses the Azure Identity library's DefaultAzureCredential which automatically
    tries multiple authentication methods including environment variables, managed identity,
    and Azure CLI.

    See https://learn.microsoft.com/en-us/azure/developer/python/sdk/authentication/credential-chains
    for more details on DefaultAzureCredential.
    """

    DEFAULT_SCOPE = "https://storage.azure.com/.default"

    def __init__(
        self,
        scopes: list[str] | None = None,
        **credential_kwargs: Any,
    ):
        """
        Initialize EntraAuthManager.

        Args:
            scopes: List of OAuth2 scopes. Defaults to ["https://storage.azure.com/.default"].
            **credential_kwargs: Arguments passed to DefaultAzureCredential.
                Supported authentication methods:
                - Environment Variables: Set AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET
                - Managed Identity: Works automatically on Azure; for user-assigned, pass managed_identity_client_id
                - Azure CLI: Works automatically if logged in via `az login`
                - Workload Identity: Works automatically in AKS with workload identity configured  # codespell:ignore aks
        """
        try:
            from azure.identity import DefaultAzureCredential
        except ImportError as e:
            raise ImportError("Azure Identity library not found. Please install with: pip install pyiceberg[entra-auth]") from e

        self._scopes = scopes or [self.DEFAULT_SCOPE]
        self._lock = threading.Lock()
        self._token: str | None = None
        self._expires_at: float = 0
        self._credential = DefaultAzureCredential(**credential_kwargs)

    def _refresh_token(self) -> None:
        """Refresh the access token from Azure."""
        token = self._credential.get_token(*self._scopes)
        self._token = token.token
        # expires_on is a Unix timestamp; add a 60-second margin for safety
        self._expires_at = token.expires_on - 60

    def _get_token(self) -> str:
        """Get a valid access token, refreshing if necessary."""
        with self._lock:
            if not self._token or time.time() >= self._expires_at:
                self._refresh_token()
            if self._token is None:
                raise ValueError("Failed to obtain Entra access token")
            return self._token

    def auth_header(self) -> str:
        """Return the Authorization header value with a valid Bearer token."""
        return f"Bearer {self._get_token()}"

```

--------------------------------

### Create and Append to PyIceberg Table

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Demonstrates creating an in-memory catalog, creating a table, appending data using PyArrow, and registering it with DataFusion.

```python
from datafusion import SessionContext
from pyiceberg.catalog import load_catalog
import pyarrow as pa
catalog = load_catalog("catalog", type="in-memory")
catalog.create_namespace_if_not_exists("default")
data = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]})
iceberg_table = catalog.create_table("default.test", schema=data.schema)
iceberg_table.append(data)
ctx = SessionContext()
ctx.register_table("test", iceberg_table)
ctx.table("test").show()
```

--------------------------------

### get()

Source: https://py.iceberg.apache.org/reference/pyiceberg/cli/console

Fetch properties on tables/namespaces.

```APIDOC
## get()

### Description
Fetch properties on tables/namespaces.

### Method
Not applicable (CLI command group)

### Endpoint
Not applicable (CLI command group)
```

--------------------------------

### __init__

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/rest

Initializes the Rest Catalog. It requires a catalog name and can accept additional properties for configuration, such as authentication credentials or a URI.

```APIDOC
## __init__ (name, **properties)

### Description
Initializes the Rest Catalog. You either need to provide a client_id and client_secret, or an already valid token.

### Parameters
#### Path Parameters
- **name** (str) - Required - Name to identify the catalog.
- **properties** (str) - Optional - Properties that are passed along to the configuration. Defaults to `{}`.
```

--------------------------------

### Upload Release to PyPI using Twine

Source: https://py.iceberg.apache.org/how-to-release

Checks out the Apache SVN repository, navigates to the release directory, and uploads the built distribution files to PyPI using `twine`. Ensure `twine` is installed and potentially configured with an API token.

```bash
svn checkout https://dist.apache.org/repos/dist/release/iceberg /tmp/iceberg-dist-release/
cd /tmp/iceberg-dist-release/pyiceberg-${VERSION}
twine upload pyiceberg-*.whl pyiceberg-*.tar.gz
```

--------------------------------

### NotInstalledError

Source: https://py.iceberg.apache.org/reference/pyiceberg/exceptions

Raised when an optional dependency is not installed.

```python
class NotInstalledError(Exception):
    """When an optional dependency is not installed."""
```

--------------------------------

### Get Table History

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Retrieves the snapshot history of the table.

```python
def history(self) -> list[SnapshotLogEntry]:
    """Get the snapshot history of this table."""
    return self.metadata.snapshot_log
```

--------------------------------

### Initialize PyIceberg REST Catalog

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog/rest

Initializes a REST Catalog instance. Requires either client credentials or a valid token. The catalog URI and other properties are passed during initialization.

```python
def __init__(self, name: str, **properties: str):
    """Rest Catalog.

    You either need to provide a client_id and client_secret, or an already valid token.

    Args:
        name: Name to identify the catalog.
        properties: Properties that are passed along to the configuration.
    """
    super().__init__(name, **properties)
    self._auth_manager: AuthManager | None = None
    self.uri = properties[URI]
    self._fetch_config()
    self._session = self._create_session()
```

--------------------------------

### Get All Snapshots

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Returns a list of all snapshots associated with the table.

```python
def snapshots(self) -> list[Snapshot]:
    return self.metadata.snapshots
```

--------------------------------

### FsspecInputFile Constructor

Source: https://py.iceberg.apache.org/reference/pyiceberg/io/fsspec

Initializes an FsspecInputFile instance with a location URI and an fsspec filesystem instance.

```APIDOC
## FsspecInputFile

Bases: `InputFile`

An input file implementation for the FsspecFileIO.

### Parameters

- **location** (str): A URI to a file location. (Required)
- **fs** (`AbstractFileSystem`): An fsspec filesystem instance. (Required)
```

--------------------------------

### Get Table Properties

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Retrieves the properties of the table as a dictionary.

```python
@property
def properties(self) -> dict[str, str]:
    """Properties of the table."""
    return self.metadata.properties
```

--------------------------------

### Get Table Schema

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Retrieves the current schema for the table.

```python
def schema(self) -> Schema:
    """Return the schema for this table."""
    return next(schema for schema in self.metadata.schemas if schema.schema_id == self.metadata.current_schema_id)
```

--------------------------------

### Get Snapshot References

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Retrieves all snapshot references associated with the table.

```python
def refs(self) -> dict[str, SnapshotRef]:
    """Return the snapshot references in the table."""
    return self.metadata.refs
```

--------------------------------

### Server-Side File Planning in PyIceberg

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

This snippet demonstrates how to plan files using server-side scan planning with PyIceberg. It requires a RestCatalog and a table identifier, then constructs a PlanTableScanRequest to send to the catalog.

```python
    def _plan_files_server_side(self) -> Iterable[FileScanTask]:
        """Plan files using REST server-side scan planning."""
        from pyiceberg.catalog.rest import RestCatalog
        from pyiceberg.catalog.rest.scan_planning import PlanTableScanRequest

        if not isinstance(self.catalog, RestCatalog):
            raise TypeError("REST scan planning requires a RestCatalog")
        if self.table_identifier is None:
            raise ValueError("REST scan planning requires a table identifier")

        request = PlanTableScanRequest(
            snapshot_id=self.snapshot_id,
            select=list(self.selected_fields) if self.selected_fields != ("*",) else None,
            filter=self.row_filter if self.row_filter != ALWAYS_TRUE else None,
            case_sensitive=self.case_sensitive,
        )

        return self.catalog.plan_scan(self.table_identifier, request)
```

--------------------------------

### Get Name Mapping

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Retrieves the table's field-id NameMapping.

```python
def name_mapping(self) -> NameMapping | None:
    """Return the table's field-id NameMapping."""
    return self.metadata.name_mapping()
```

--------------------------------

### Create Table with PyArrow Schema

Source: https://py.iceberg.apache.org/api

Creates a table named 'docs_example.bids' using a PyArrow schema definition.

```python
import pyarrow as pa

schema = pa.schema([
        pa.field("foo", pa.string(), nullable=True),
        pa.field("bar", pa.int32(), nullable=False),
        pa.field("baz", pa.bool_(), nullable=True),
])

catalog.create_table(
    identifier="docs_example.bids",
    schema=schema,
)

```

--------------------------------

### Configure In-Memory Catalog

Source: https://py.iceberg.apache.org/configuration

Sets up an in-memory catalog using SQLite for testing and demos. Specify the warehouse directory for data storage.

```yaml
catalog:
  default:
    type: in-memory
    warehouse: /tmp/pyiceberg/warehouse

```

--------------------------------

### Table.current_snapshot

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Gets the current snapshot for this table, or None if there is no current snapshot.

```APIDOC
## Table.current_snapshot

### Description
Get the current snapshot for this table, or None if there is no current snapshot.

### Returns
* Snapshot | None - The current snapshot object or None.
```

--------------------------------

### Get Table Location

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Returns the base location URI of the table.

```python
def location(self) -> str:
    """Return the table's base location."""
    return self.metadata.location
```

--------------------------------

### Initialize AvroOutputFile

Source: https://py.iceberg.apache.org/reference/pyiceberg/avro/file

Initializes an AvroOutputFile with the necessary components for writing Avro data. It sets up the output file, schema, and writer, and generates sync bytes.

```python
class AvroOutputFile(Generic[D]):
    output_file: OutputFile
    output_stream: OutputStream
    file_schema: Schema
    schema_name: str
    encoder: BinaryEncoder
    sync_bytes: bytes
    writer: Writer

    def __init__(
        self,
        output_file: OutputFile,
        file_schema: Schema,
        schema_name: str,
        record_schema: Schema | None = None,
        metadata: dict[str, str] = EMPTY_DICT,
    ) -> None:
        self.output_file = output_file
        self.file_schema = file_schema
        self.schema_name = schema_name
        self.sync_bytes = os.urandom(SYNC_SIZE)
        self.writer = (
            construct_writer(file_schema=self.file_schema)
            if record_schema is None
            else resolve_writer(record_schema=record_schema, file_schema=self.file_schema)
        )
        self.metadata = metadata
```

--------------------------------

### Get Table Schema

Source: https://py.iceberg.apache.org/reference/pyiceberg/cli/console

Retrieves and displays the schema of a specified table.

```python
@run.command()
@click.argument("identifier")
@click.pass_context
@catch_exception()
def schema(ctx: Context, identifier: str) -> None:
    """Get the schema of the table."""
    catalog, output = _catalog_and_output(ctx)
    table = catalog.load_table(identifier)
    output.schema(table.schema())
```

--------------------------------

### Upgrade Pip

Source: https://py.iceberg.apache.org/

Ensure you are using an up-to-date version of pip before installing PyIceberg.

```bash
pip install --upgrade pip

```

--------------------------------

### __enter__

Source: https://py.iceberg.apache.org/reference/pyiceberg/catalog

Enters the catalog's context manager, returning the catalog instance for use within a 'with' statement.

```APIDOC
## __enter__()

### Description
Enter the context manager.

### Returns
- **Catalog** (`Catalog`): The catalog instance.
```

--------------------------------

### Table.snapshot_by_id

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Gets the snapshot of this table with the given ID, or None if there is no matching snapshot.

```APIDOC
## Table.snapshot_by_id

### Description
Get the snapshot of this table with the given id, or None if there is no matching snapshot.

### Parameters
#### Path Parameters
* **snapshot_id** (int) - Required - The ID of the snapshot to retrieve.

### Returns
* Snapshot | None - The Snapshot object with the matching ID, or None.
```

--------------------------------

### Get Table Sort Order

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Retrieves the default sort order for the table.

```python
def sort_order(self) -> SortOrder:
    """Return the sort order of this table."""
    return next(
        sort_order for sort_order in self.metadata.sort_orders if sort_order.order_id == self.metadata.default_sort_order_id
    )
```

--------------------------------

### Load and configure a local SQL catalog

Source: https://py.iceberg.apache.org/

Open a Python REPL to load and configure a local SQL catalog using SQLite for storing metadata and the local filesystem for data files. This setup is suitable for local testing.

```python
from pyiceberg.catalog import load_catalog

warehouse_path = "/tmp/warehouse"
catalog = load_catalog(
    "default",
    **{
        'type': 'sql',
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)

```

--------------------------------

### Get Table Partition Spec

Source: https://py.iceberg.apache.org/reference/pyiceberg/table

Retrieves the default partition specification for the table.

```python
def spec(self) -> PartitionSpec:
    """Return the partition spec of this table."""
    return next(spec for spec in self.metadata.partition_specs if spec.spec_id == self.metadata.default_spec_id)
```

--------------------------------

### Get Table Location

Source: https://py.iceberg.apache.org/reference/pyiceberg/cli/console

Retrieves and displays the storage location of a specified table.

```python
@run.command()
@click.argument("identifier")
@click.pass_context
@catch_exception()
def location(ctx: Context, identifier: str) -> None:
    """Return the location of the table."""
    catalog, output = _catalog_and_output(ctx)
    table = catalog.load_table(identifier)
    output.text(table.location())
```
Tokens:
Raw