### Install VAST DB Python SDK

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This command installs the VAST DB Python SDK using pip, the standard Python package installer. It ensures all necessary dependencies are downloaded and configured for use.

```bash
pip install vastdb
```

--------------------------------

### Connect, Create Schema/Table, Insert, and Select Data in VAST DB

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This Python example demonstrates how to establish a connection to VAST DB, create a schema and a table with specified columns, insert data from a PyArrow table, and then select all data to verify the insertion. It highlights basic CRUD operations within a transaction context.

```python
import pyarrow as pa
import vastdb

session = vastdb.connect(
    endpoint='http://vip-pool.v123-xy.VastENG.lab',
    access=AWS_ACCESS_KEY_ID,
    secret=AWS_SECRET_ACCESS_KEY)

with session.transaction() as tx:
    bucket = tx.bucket("bucket-name")

    schema = bucket.create_schema("schema-name")
    print(bucket.schemas())

    columns = pa.schema([
        ('c1', pa.int16()),
        ('c2', pa.float32()),
        ('c3', pa.utf8()),
    ])
    table = schema.create_table("table-name", columns)
    print(schema.tables())
    print(table.columns())

    arrow_table = pa.table(schema=columns, data=[
        [111, 222, 333],
        [0.5, 1.5, 2.5],
        ['a', 'bb', 'ccc'],
    ])
    table.insert(arrow_table)

    # run `SELECT * FROM t`
    reader = table.select()  # return a `pyarrow.RecordBatchReader`
    result = reader.read_all()  # build an PyArrow Table from the `pyarrow.RecordBatch` objects read from VAST
    assert result == arrow_table

    # the transaction is automatically committed when exiting the context
```

--------------------------------

### Integrating VAST Database Data with DuckDB for SQL Queries

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This example shows how to use DuckDB to post-process PyArrow record batches retrieved from the VAST Database. It establishes a DuckDB connection, selects specific columns with a predicate, and executes a SQL query on the batches. A crucial note is that the VAST DB transaction must remain active while the DuckDB query is executing and fetching results.

```python
from ibis import _

import duckdb
conn = duckdb.connect()

with session.transaction() as tx:
    table = tx.bucket("bucket-name").schema("schema-name").table("table-name")
    batches = table.select(columns=['c1'], predicate=(_.c2 > 2))
    print(conn.execute("SELECT sum(c1) FROM batches").arrow())
```

--------------------------------

### Apply Filters and Projections in VAST DB Queries

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This Python snippet illustrates how to use predicate and projection pushdown for efficient data querying in VAST DB. It shows examples of filtering data based on conditions (e.g., greater than, between, contains) and selecting specific columns, leveraging the `ibis` library for predicate construction.

```python
    from ibis import _

    # SELECT c1 FROM t WHERE (c2 > 2) AND (c3 IS NULL)
    table.select(columns=['c1'],
                 predicate=(_.c2 > 2) & _.c3.isnull())

    # SELECT c2, c3 FROM t WHERE (c2 BETWEEN 0 AND 1) OR (c2 > 10)
    table.select(columns=['c2', 'c3'],
                 predicate=(_.c2.between(0, 1) | (_.c2 > 10))

    # SELECT * FROM t WHERE c3 LIKE '%substring%'
    table.select(predicate=_.c3.contains('substring'))
```

--------------------------------

### Manage Semi-sorted Projections in VAST DB

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This Python example illustrates how to create, list, and delete semi-sorted projections in VAST DB. Projections can optimize query performance by pre-sorting data based on specified columns, enhancing efficiency for common query patterns.

```python
p = table.create_projection('proj', sorted=['c3'], unsorted=['c1'])
print(table.projections())
print(p.get_stats())
p.drop()
```

--------------------------------

### Import Single Parquet File to VAST DB via S3

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This Python example demonstrates how to import a single Parquet file into a VAST DB table using the S3 protocol. It involves writing a PyArrow table to a temporary Parquet file, uploading it to an S3 bucket, and then creating a VAST DB table from that file without client-side copying.

```python
    with tempfile.NamedTemporaryFile() as f:
        pa.parquet.write_table(arrow_table, f.name)
        s3.put_object(Bucket='bucket-name', Key='staging/file.parquet', Body=f)

    schema = tx.bucket('bucket-name').schema('schema-name')
    table = util.create_table_from_files(
        schema=schema, table_name='imported-table',
        parquet_files=['/bucket-name/staging/file.parquet'])
```

--------------------------------

### Initialize and Use QueryConfig for Table Selection

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/config.md

Demonstrates how to import `QueryConfig`, create an instance with default values, and then pass it to the `table.select()` method to apply custom query configurations.

```python
from vastdb.config import QueryConfig
cfg = QueryConfig()  # default configuration values
# <modify configuration>
table.select(columns=['c1'], predicate=(_.c2 > 2), config=cfg)
```

--------------------------------

### Initialize and Apply QueryConfig for Table Selection

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/source/docs/config.md

This snippet demonstrates the basic usage of `QueryConfig` by initializing it and passing it to the `table.select()` method. It shows how to apply custom configuration values to a table selection operation.

```python
from vastdb.config import QueryConfig
cfg = QueryConfig()  # default configuration values
# <modify configuration>
table.select(columns=['c1'], predicate=(_.c2 > 2), config=cfg)
```

--------------------------------

### Python Project Dependencies List

Source: https://github.com/vast-data/vastdb_sdk/blob/main/requirements-dev.txt

This snippet details the Python packages required for the `vastdb_sdk` project. It includes essential libraries for data handling, cloud interaction, testing, and code quality.

```Python
-r requirements.txt

black
boto3
coverage
duckdb
exchange-calendars
fsspec
ipython
mypy
pandas-stubs
pytest
pytest-cov
pytest-retry
pytest-xdist
ruff
twine
typer
```

--------------------------------

### Configure Multiple Data Endpoints for Query Splitting

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/config.md

Illustrates how to explicitly list multiple CNode URLs (VIPs or domain names) in `QueryConfig.data_endpoints` to distribute the scanning work of a query across several CNode connections, enhancing parallel processing.

```python
cfg.data_endpoints=[
    "http://172.19.196.1",
    "http://172.19.196.2",
    "http://172.196.196.3",
    "http://172.196.196.4"
]
```

--------------------------------

### Accessing VAST Database Snapshots with Python SDK

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This snippet demonstrates how to list and query VAST Database snapshots using the Python SDK. It retrieves the first available snapshot and then selects data from a specified table within a schema.

```python
snaps = bucket.list_snapshots()
batches = snaps[0].schema('schema-name').table('table-name').select()
```

--------------------------------

### Querying the VAST Catalog with Python SDK

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This snippet demonstrates how to connect to the VAST Database and query the VAST Catalog as a regular table. It retrieves 'element_type' information, converts the result to a Pandas DataFrame, and then calculates the total number of elements, the count of files/objects, and lists the distinct element types present in the system.

```python
import pyarrow as pa
import vastdb

session = vastdb.connect(
    endpoint='http://vip-pool.v123-xy.VastENG.lab',
    access=AWS_ACCESS_KEY_ID,
    secret=AWS_SECRET_ACCESS_KEY)

with session.transaction() as tx:
    table = tx.catalog().select(['element_type']).read_all()
    df = table.to_pandas()

    total_elements = len(df)
    print(f"Total elements in the catalog: {total_elements}")

    file_count = (df['element_type'] == 'FILE').sum()
    print(f"Number of files/objects: {file_count}")

    distinct_elements = df['element_type'].unique()
    print("Distinct element types on the system:")
    print(distinct_elements)
```

--------------------------------

### vastdb.schema Module API Reference Generation

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/source/schema.rst

API documentation for the `vastdb.schema` Python module, generated via Sphinx's `automodule` directive. This directive ensures all members, including undocumented ones, and inheritance relationships are included in the generated documentation, providing a complete API specification.

```reStructuredText
.. automodule:: vastdb.schema
   :members:
   :undoc-members:
   :show-inheritance:
```

--------------------------------

### Exporting PyArrow Record Batches to Parquet File

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This code illustrates how to export a stream of PyArrow record batches, obtained from a VAST Database table selection, directly into a Parquet file. It utilizes `contextlib.closing` and `pa.parquet.ParquetWriter` for efficient data writing.

```python
batches = table.select()
with contextlib.closing(pa.parquet.ParquetWriter('/path/to/file.parquet', batches.schema)) as writer:
    for batch in table_batches:
        writer.write_batch(batch)
```

--------------------------------

### Configure Multiple Data Endpoints for Query Splitting

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/source/docs/config.md

This snippet illustrates how to specify multiple CNode data endpoints within a `QueryConfig` object. This configuration allows the system to distribute the scanning work of a query across several CNode connections, enhancing performance for large-scale queries.

```python
cfg.data_endpoints=[
    "http://172.19.196.1",
    "http://172.19.196.2",
    "http://172.19.196.3",
    "http://172.196.196.4"
]
```

--------------------------------

### VAST Database Predicate Pushdown Structure and Supported Operators

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/source/docs/predicate.md

Defines the logical structure for filter predicates supported by VAST Database for pushdown, including AND/OR combinations and specific operators for column predicates. It also lists the Arrow types that support predicate pushdown.

```APIDOC
Predicate Structure:
  - Top-level: An AND between multiple column predicates (e.g., p = p1 & p2 & p3)
  - Column-level: An OR between range predicates (e.g., ci = r1 | r2 | r3 | r4)

Range Predicate Operators (where "x" is a column name, and c is a constant):
  - Comparison operators:
    - t["x"] < c
    - t["x"] <= c
    - t["x"] == c
    - t["x"] > c
    - t["x"] >= c
    - t["x"] != c
  - Membership operator:
    - t["x"].isin([c1, c2, c3, c4, c5])
  - NULL checking:
    - t["x"].isnull()
    - ~t["x"].isnull()
  - String matching:
    - t["x"].startswith("prefix")
    - t["x"].contains("substr")

Supported Arrow Column Types for Predicate Pushdown:
  - int8
  - int16
  - int32
  - int64
  - float32
  - float64
  - utf8
  - bool
  - decimal128
  - binary
  - date32
  - time32
  - time64
  - timestamp
```

--------------------------------

### Sphinx Automodule Directive for vastdb.transaction

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/source/transaction.rst

This reStructuredText snippet uses the Sphinx `automodule` directive to automatically generate comprehensive API documentation for the `vastdb.transaction` Python module. It ensures that all members, including those without explicit docstrings, are documented, and their class inheritance is displayed.

```RST
.. automodule:: vastdb.transaction
   :members:
   :undoc-members:
   :show-inheritance:
```

--------------------------------

### VAST Database Predicate Pushdown Structure and Supported Types

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/predicate.md

Defines the logical structure for filter predicates supported by VAST Database for pushdown, including how column predicates are combined and the various range operators available. It also lists the Arrow column types that are compatible with predicate pushdown.

```APIDOC
Predicate Structure:
  - Overall Predicate 'p': Logical AND of multiple column predicates (e.g., p = p1 & p2 & p3)
  - Column Predicate 'pi': Logical OR of multiple range predicates (e.g., ci = r1 | r2 | r3 | r4)

Range Predicate 'rj' Operators (where "x" is a column name, and 'c' is a constant):
  - Comparison Operators:
    - t["x"] < c
    - t["x"] <= c
    - t["x"] == c
    - t["x"] > c
    - t["x"] >= c
    - t["x"] != c
  - is_in Operator:
    - t["x"].isin([c1, c2, c3, c4, c5])
  - NULL Checking:
    - t["x"].isnull()
    - ~t["x"].isnull()
  - Prefix Match:
    - t["x"].startswith("prefix")
  - Substring Match:
    - t["x"].contains("substr")

Supported Arrow Types for Predicate Pushdown:
  - int8
  - int16
  - int32
  - int64
  - float32
  - float64
  - utf8
  - bool
  - decimal128
  - binary
  - date32
  - time32
  - time64
  - timestamp
```

--------------------------------

### Import Multiple Parquet Files Concurrently to VAST DB via S3

Source: https://github.com/vast-data/vastdb_sdk/blob/main/README.md

This Python snippet shows how to efficiently import multiple Parquet files into a VAST DB table concurrently, leveraging multiple CNodes' cores. It demonstrates creating a table from a list of S3-pathed Parquet files, optimizing for large-scale data ingestion.

```python
    schema = tx.bucket('bucket-name').schema('schema-name')
    table = util.create_table_from_files(
        schema=schema, table_name='large-imported-table',
        parquet_files=[f'/bucket-name/staging/file{i}.parquet' for i in range(10)])
```

--------------------------------

### Set Internal CNode Concurrency for Selective Queries

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/source/docs/config.md

This snippet demonstrates how to adjust the `num_sub_splits` parameter in `QueryConfig`. Increasing this value allows a single RPC request to be processed by multiple CNode cores, which can significantly improve the performance of selective queries.

```python
cfg.num_sub_splits = 8
```

--------------------------------

### Set Internal CNode Concurrency for Selective Queries

Source: https://github.com/vast-data/vastdb_sdk/blob/main/docs/config.md

Shows how to configure `QueryConfig.num_sub_splits` to control the number of worker threads a CNode spawns per RPC. Increasing this value can help process a single request from multiple CNode cores, particularly benefiting selective queries.

```python
cfg.num_sub_splits = 8
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.