Typedspark

https://typedspark.readthedocs.io/en/latest/
Admin
Typedspark provides column-wise type annotations for PySpark DataFrames, enhancing readability,...
Tokens:44,196
Snippets:474
Trust Score:9.6
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Typedspark

Source: https://typedspark.readthedocs.io/en/latest/README.html

Install the typedspark library using pip. This is the basic installation command.

```bash
pip install typedspark

```

--------------------------------

### Set Up Development Environment

Source: https://typedspark.readthedocs.io/en/latest/contributing.html

Installs Python 3.11, creates a virtual environment named 'typedspark', activates it, and installs project dependencies. Requires pyenv to be installed.

```bash
pyenv install 3.11
pyenv virtualenv 3.11 typedspark
pyenv activate typedspark
pip install -r requirements.txt
pip install -r requirements-dev.txt

```

--------------------------------

### Set Up Development Environment

Source: https://typedspark.readthedocs.io/en/latest/_sources/contributing.rst.txt

Installs Python 3.11, creates a virtual environment named 'typedspark', activates it, and installs project dependencies.

```bash
pyenv install 3.11
pyenv virtualenv 3.11 typedspark
pyenv activate typedspark
pip install -r requirements.txt
pip install -r requirements-dev.txt
```

--------------------------------

### Install Pre-commit Hook

Source: https://typedspark.readthedocs.io/en/latest/_sources/contributing.rst.txt

Installs the pre-commit hook to run checks on code before committing. Requires Spark to be set up.

```bash
pre-commit install
```

--------------------------------

### Install Pre-commit Hook

Source: https://typedspark.readthedocs.io/en/latest/contributing.html

Installs the pre-commit hook to run checks on code before committing. This command should be run from the repository root.

```bash
pre-commit install

```

--------------------------------

### Install Typedspark with PySpark Dependency

Source: https://typedspark.readthedocs.io/en/latest/README.html

Install typedspark along with its PySpark dependency. Use this if your environment does not have PySpark pre-installed.

```bash
pip install "typedspark[pyspark]"

```

--------------------------------

### Example Unit Test with TypedSpark

Source: https://typedspark.readthedocs.io/en/latest/create_empty_datasets.html

Demonstrates a full unit test using typedspark, including creating a partially filled dataset, applying a transformation, and asserting equality with an expected dataset.

```python
from pyspark.sql import SparkSession
from pyspark.sql.types import LongType, StringType
from typedspark import Column, DataSet, Schema, create_partially_filled_dataset, transform_to_schema
from chispa.dataframe_comparer import assert_df_equality


class Person(Schema):
    name: Column[StringType]
    age: Column[LongType]


def birthday(df: DataSet[Person]) -> DataSet[Person]:
    return transform_to_schema(df, Person, {Person.age: Person.age + 1})


def test_birthday(spark: SparkSession):
    df = create_partially_filled_dataset(
        spark,
        Person,
        {
            Person.name: ["Alice", "Bob"],
            Person.age: [20, 30],
        },
    )

    observed = birthday(df)
    expected = create_partially_filled_dataset(
        spark,
        Person,
        {
            Person.name: ["Alice", "Bob"],
            Person.age: [21, 31],
        },
    )

    assert_df_equality(observed, expected, ignore_row_order=True, ignore_nullable=True)
```

--------------------------------

### Get schema name

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns the name used to initialize the schema.

```python
get_schema_name()
```

--------------------------------

### Define a Schema for a Job

Source: https://typedspark.readthedocs.io/en/latest/api.html

Example of defining a schema for a Job with position and salary using Column types.

```python
class Job(Schema):
    position: Column[StringType]
    salary: Column[LongType]
```

--------------------------------

### Traditional DataSet Transformation Example

Source: https://typedspark.readthedocs.io/en/latest/_sources/dataset_implements.ipynb.txt

Illustrates a transformation function `birthday` using `DataSet` that requires explicit schema typing, which can become cumbersome with many schemas.

```python
from typing import TypeVar, Union

from typedspark import DataSet, transform_to_schema

T = TypeVar("T", bound=Union[Person, Pet])


def birthday(df: DataSet[T]) -> DataSet[T]:
    return transform_to_schema(
        df,
        df.typedspark_schema,
        {Person.age: Person.age + 1},
    )
```

--------------------------------

### Get DLT kwargs for a schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates a representation of the Schema suitable for use with Delta Live Tables (DLT) configurations.

```python
@dlt.table(**DimPatient.get_dlt_kwargs())
def table_definition() -> DataSet[DimPatient]:
    <your table definition here>
```

--------------------------------

### Unit Test Example with TypedSpark

Source: https://typedspark.readthedocs.io/en/latest/_sources/create_empty_datasets.ipynb.txt

Demonstrates a unit test for a 'birthday' transformation function using TypedSpark. It creates a dataset, applies the transformation, and asserts equality with an expected DataFrame.

```python
from pyspark.sql import SparkSession
from pyspark.sql.types import LongType, StringType
from typedspark import Column, DataSet, Schema, create_partially_filled_dataset, transform_to_schema
from chispa.dataframe_comparer import assert_df_equality


class Person(Schema):
    name: Column[StringType]
    age: Column[LongType]


def birthday(df: DataSet[Person]) -> DataSet[Person]:
    return transform_to_schema(df, Person, {Person.age: Person.age + 1})


def test_birthday(spark: SparkSession):
    df = create_partially_filled_dataset(

```

--------------------------------

### Column.full_path

Source: https://typedspark.readthedocs.io/en/latest/api.html

Gets the full path of a Column.

```APIDOC
## Column.full_path

### Description
Gets the full path of a Column.

### Method
`Column.full_path`

### Parameters
None

### Response
#### Success Response
- **full_path** (string) - The full path of the column.
```

--------------------------------

### Create Sample DataFrame with TypedSpark Schema

Source: https://typedspark.readthedocs.io/en/latest/_sources/create_schema_in_notebook.ipynb.txt

Defines a `Vaccinations` schema using TypedSpark and creates a DataFrame from a dictionary. This example shows how to populate a DataFrame with specific data types and column names.

```python
from datetime import timedelta, datetime
from pyspark.sql.functions import first
from pyspark.sql.types import LongType, StringType, DateType
from typedspark import Column, Schema, create_partially_filled_dataset, create_schema

date = datetime(2023, 10, 2)


class Vaccinations(Schema):
    vaccination_id: Column[LongType]
    pet_id: Column[LongType]
    vaccine_name: Column[StringType]
    vaccine_date: Column[DateType]
    next_due_date: Column[DateType]


vaccinations = create_partially_filled_dataset(
    spark,
    Vaccinations,
    {
        Vaccinations.vaccination_id: [1, 2, 3, 4, 5, 6, 7],
        Vaccinations.pet_id: [1, 2, 3, 1, 3, 2, 3],
        Vaccinations.vaccine_name: [
            "rabies",
            "rabies",
            "rabies",
            "lyme",
            "lyme",
            "influenza",
            "influenza",
        ],
        Vaccinations.next_due_date: [
            date + timedelta(days=32),
            date + timedelta(days=6),
            date + timedelta(days=12),
            date + timedelta(days=15),
            date + timedelta(days=2),
            date + timedelta(days=1),
            date + timedelta(days=3),
        ],
        Vaccinations.vaccine_date: [
            date + timedelta(days=32) - timedelta(days=365),
            date + timedelta(days=6) - timedelta(days=365),
            date + timedelta(days=12) - timedelta(days=365),
            date + timedelta(days=15) - timedelta(days=365),
            date + timedelta(days=2) - timedelta(days=365),
            date + timedelta(days=1) - timedelta(days=365),
            date + timedelta(days=3) - timedelta(days=365),
        ],
    },
)
vaccinations.show()
```

--------------------------------

### Get schema definition as string

Source: https://typedspark.readthedocs.io/en/latest/api.html

Generates a string representation of the schema definition, optionally including documentation and subschemas, and generating necessary imports.

```python
get_schema_definition_as_string(_schema_name : str | None = None_, _include_documentation : bool = False_, _generate_imports : bool = True_, _add_subschemas : bool = True_) -> str
```

--------------------------------

### Define a dynamically loaded schema

Source: https://typedspark.readthedocs.io/en/latest/loading_datasets_in_notebooks.html

Example of defining a schema class when the schema name is not explicitly provided during loading.

```python
from pyspark.sql.types import LongType, StringType

from typedspark import Column, Schema


class DynamicallyLoadedSchema(Schema):
    name: Column[StringType]
    age: Column[LongType]
```

--------------------------------

### Resolve Ambiguity with Schema Registration

Source: https://typedspark.readthedocs.io/en/latest/_sources/column_ambiguity.ipynb.txt

This example demonstrates how to resolve ambiguous column references by registering schemas to DataFrames before performing a join. It uses `register_schema_to_dataset` to associate 'Person' and 'Job' schemas with their respective DataFrames, allowing a successful join on the 'id' column.

```python
from typedspark import register_schema_to_dataset

person = register_schema_to_dataset(df_a, Person)
job = register_schema_to_dataset(df_b, Job)
(
    df_a.join(
        df_b,
        person.id == job.id,
    ).show()
)
```

--------------------------------

### Define and Create StructType Data

Source: https://typedspark.readthedocs.io/en/latest/_sources/structtypes_in_notebooks.ipynb.txt

Defines nested schemas using typedspark's Schema and StructType, creates a partially filled dataset, and registers it as a temporary view. This snippet sets up the example data for demonstrating StructType handling.

```python
from typedspark import Schema, Column, StructType, create_partially_filled_dataset, load_table
from pyspark.sql.types import IntegerType


class Values(Schema):
    a: Column[IntegerType]
    b: Column[IntegerType]


class Container(Schema):
    values: Column[StructType[Values]]


create_partially_filled_dataset(
    spark,
    Container,
    {
        Container.values: create_partially_filled_dataset(
            spark,
            Values,
            {Values.a: [1, 2, 3]},
        ).collect(),
    },
).createOrReplaceTempView("structtype_table")

container, ContainerSchema = load_table(spark, "structtype_table", "Container")
```

--------------------------------

### Build Documentation Locally

Source: https://typedspark.readthedocs.io/en/latest/_sources/contributing.rst.txt

Cleans the existing documentation and builds the HTML version locally. The output can be found at 'docs/build/html/index.html'.

```bash
cd docs/; make clean; make html; cd ..
```

--------------------------------

### ColumnMeta.comment

Source: https://typedspark.readthedocs.io/en/latest/api.html

Gets the comment of ColumnMeta.

```APIDOC
## ColumnMeta.comment

### Description
Gets the comment of ColumnMeta.

### Method
`ColumnMeta.comment`

### Parameters
None

### Response
#### Success Response
- **comment** (string) - The comment associated with the column metadata.
```

--------------------------------

### Create a Partially Filled TypedSpark DataSet

Source: https://typedspark.readthedocs.io/en/latest/advanced_linting_support.html

Demonstrates creating a TypedSpark DataSet with a defined schema 'A' and populating it with data. This is useful for testing and demonstrating schema-aware operations.

```python
from typedspark import Column, Schema, DataSet, create_partially_filled_dataset
from pyspark.sql.types import StringType


class A(Schema):
    a: Column[StringType]


df = create_partially_filled_dataset(
    spark,
    A,
    {
        A.a: ["a", "b", "c"],
    },
)
res = df.filter(A.a == "a")

```

--------------------------------

### StructType.schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Gets the schema definition from a StructType.

```APIDOC
## StructType.schema

### Description
Gets the schema definition from a StructType.

### Method
`StructType.schema`

### Parameters
None

### Response
#### Success Response
- **schema** (StructType) - The schema definition.
```

--------------------------------

### Run Documentation Notebooks

Source: https://typedspark.readthedocs.io/en/latest/_sources/contributing.rst.txt

Executes all documentation notebooks in the 'docs/' directory and strips metadata for manageable diffs in pull requests.

```bash
sh docs/run_notebooks.sh
```

--------------------------------

### DataSetImplements.typedspark_schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Gets the Typedspark schema of the DataSetImplements.

```APIDOC
## DataSetImplements.typedspark_schema

### Description
Gets the Typedspark schema of the DataSetImplements.

### Method
`DataSetImplements.typedspark_schema`

### Parameters
None

### Response
#### Success Response
- **typedspark_schema** (Schema) - The Typedspark schema.
```

--------------------------------

### Run Documentation Notebooks

Source: https://typedspark.readthedocs.io/en/latest/contributing.html

Executes all documentation notebooks located in the 'docs/' directory and strips metadata for manageable PR diffs. Run from the repository root.

```bash
sh docs/run_notebooks.sh

```

--------------------------------

### Column.dtype

Source: https://typedspark.readthedocs.io/en/latest/api.html

Gets the data type of a Column.

```APIDOC
## Column.dtype

### Description
Gets the data type of a Column.

### Method
`Column.dtype`

### Parameters
None

### Response
#### Success Response
- **dtype** (string) - The data type of the column.
```

--------------------------------

### sample

Source: https://typedspark.readthedocs.io/en/latest/api.html

Samples data from the DataSet. This method accepts arbitrary positional and keyword arguments.

```APIDOC
## sample

### Description
Samples data from the DataSet. This method accepts arbitrary positional and keyword arguments.

### Method
`sample`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **DataSet[_Implementation]** - The sampled DataSet.

#### Response Example
None
```

--------------------------------

### sampleBy

Source: https://typedspark.readthedocs.io/en/latest/api.html

Samples data from the DataSet based on column fractions.

```APIDOC
## sampleBy

### Description
Samples data from the DataSet based on column fractions.

### Method
`sampleBy`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **DataSet[_Implementation]** - The sampled DataSet.

#### Response Example
None
```

--------------------------------

### Get schema docstring

Source: https://typedspark.readthedocs.io/en/latest/api.html

Retrieves the docstring associated with a schema definition.

```python
get_docstring() -> str | None
```

--------------------------------

### DataSetImplements.sample()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a new DataSetImplements containing a random sample of this DataSetImplements.

```APIDOC
## DataSetImplements.sample()

### Description
Returns a new DataSetImplements containing a random sample of this DataSetImplements.

### Method
`DataSetImplements.sample(withReplacement: bool, fraction: float, seed: int = None)`

### Parameters
- **withReplacement** (bool) - Required - Whether to sample with replacement.
- **fraction** (float) - Required - The fraction of rows to sample.
- **seed** (int) - Optional - The seed for the random number generator.

### Response
#### Success Response
- **DataSetImplements** - A new DataSetImplements containing the random sample.
```

--------------------------------

### sample

Source: https://typedspark.readthedocs.io/en/latest/api.html

Samples data from the DataSet. This method accepts arbitrary arguments and keyword arguments for flexible sampling configurations.

```APIDOC
## sample

### Description
Randomly samples data from the DataSet. The sampling behavior can be controlled via various arguments and keyword arguments.

### Method
N/A (SDK method)

### Endpoint
N/A (SDK method)

### Parameters
- **_* args** - Variable positional arguments for sampling configuration.
- **_**kwargs** - Variable keyword arguments for sampling configuration.

### Request Example
N/A (SDK method)

### Response
#### Success Response
- **DataSet[_Schema]** - A new DataSet containing the sampled data.
```

--------------------------------

### Get schema metadata

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a dictionary containing metadata for each column within the schema.

```python
get_metadata() -> dict[str, dict[str, Any]]
```

--------------------------------

### Get Spark StructType from schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates and returns the corresponding Spark StructType for the defined schema.

```python
get_structtype() -> StructType
```

--------------------------------

### DataSetImplements.hint()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Provides a hint to the optimizer.

```APIDOC
## DataSetImplements.hint()

### Description
Provides a hint to the optimizer.

### Method
`DataSetImplements.hint(name: str, value: list[str])`

### Parameters
- **name** (str) - Required - The name of the hint.
- **value** (list[str]) - Required - The value(s) for the hint.

### Response
#### Success Response
- **DataSetImplements** - The DataSetImplements with the applied hint.
```

--------------------------------

### Get snake case name for schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Converts the schema class name into snake_case format.

```python
get_snake_case() -> str
```

--------------------------------

### Define Schema with Integer and String Columns

Source: https://typedspark.readthedocs.io/en/latest/api.html

Illustrates defining a schema with typed columns for integers and strings.

```python
class A(Schema):
    a: Column[IntegerType]
    b: Column[StringType]
```

--------------------------------

### Define Schemas and Create Empty Datasets

Source: https://typedspark.readthedocs.io/en/latest/_sources/dataset_implements.ipynb.txt

Defines several `Schema` classes (Person, Pet, Fruit) with typed columns and creates empty `DataSet` instances for each.

```python
from pyspark.sql.types import LongType, StringType
from typedspark import (
    Schema,
    Column,
    create_empty_dataset,
)


class Person(Schema):
    name: Column[StringType]
    age: Column[LongType]
    job: Column[StringType]


class Pet(Schema):
    name: Column[StringType]
    age: Column[LongType]
    type: Column[StringType]


class Fruit(Schema):
    type: Column[StringType]


person = create_empty_dataset(spark, Person)
pet = create_empty_dataset(spark, Pet)
fruit = create_empty_dataset(spark, Fruit)
```

--------------------------------

### Define ArrayType with StringType

Source: https://typedspark.readthedocs.io/en/latest/api.html

Example of defining an ArrayType for a column containing strings within a schema.

```python
class Basket(Schema):
    items: Column[ArrayType[StringType]]
```

--------------------------------

### Get all column names from a schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a list of all column names defined within a given schema.

```python
all_column_names() -> List[str]
```

--------------------------------

### Runtime Type Checking with Valid Schema

Source: https://typedspark.readthedocs.io/en/latest/_sources/type_checking.ipynb.txt

Demonstrates successful initialization of a DataSet with a schema that matches the DataFrame. No errors are raised.

```python
import pandas as pd
from typedspark import Column, DataSet, Schema
from pyspark.sql.types import LongType, StringType


class Person(Schema):
    id: Column[LongType]
    name: Column[StringType]
    age: Column[LongType]


df = spark.createDataFrame(
    pd.DataFrame(
        dict(
            id=[1, 2, 3],
            name=["John", "Jane", "Jack"],
            age=[20, 30, 40],
        )
    )
)
# no errors raised
df = DataSet[Person](df)
df.show()
```

--------------------------------

### Load all databases and tables

Source: https://typedspark.readthedocs.io/en/latest/api.html

Initializes a Databases object to load all databases and tables available in a SparkSession. Can operate silently.

```python
class typedspark.Databases(_spark : SparkSession | None = None_, _silent : bool = False_, _catalog_name : str | None = None_)
```

--------------------------------

### Get TypedSpark Schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Retrieves the schema of a DataSet. This property provides access to the underlying schema implementation.

```python
property typedspark_schema: Type[_Implementation]
```

--------------------------------

### Define a Schema with a Nested StructType

Source: https://typedspark.readthedocs.io/en/latest/api.html

Example of defining a schema for a Person that includes a nested Job struct.

```python
class Person(Schema):
    job: Column[StructType[Job]]
```

--------------------------------

### Initialize Database Object

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Initializes the Database object to load tables from a specific database. If no database name is provided, it defaults to the default database.

```python
from typedspark import Database

db = Database()
```

--------------------------------

### Generate Schema with Documentation

Source: https://typedspark.readthedocs.io/en/latest/loading_datasets_in_notebooks.html

Generates a Python class for the schema with added documentation placeholders, useful for migrating existing tables to TypedSpark.

```python
from typing import Annotated

from pyspark.sql.types import LongType, StringType

from typedspark import Column, ColumnMeta, Schema


class PersonTable(Schema):
    """Add documentation here."""

    name: Annotated[Column[StringType], ColumnMeta(comment="")]
    age: Annotated[Column[LongType], ColumnMeta(comment="")]
```

--------------------------------

### Create a Partially Filled Dataset with a List of Dictionaries

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates a DataSet with a specified Schema, filling data using a list of dictionaries, where each dictionary represents a row.

```python
df = create_partially_filled_dataset(
    spark,
    Person,
    [
        {Person.name: "John", Person.age: 30},
        {Person.name: "Jack", Person.age: 40},
        {Person.name: "Jane", Person.age: 50},
    ]
)
```

--------------------------------

### Transform DataSet to a New Schema

Source: https://typedspark.readthedocs.io/en/latest/advanced_linting_support.html

Demonstrates using the 'transform_to_schema' utility to apply a transformation function 'foo' to a DataSet, resulting in a new DataSet conforming to schema 'B'.

```python
from typedspark import transform_to_schema
from pyspark.sql.functions import lit


class B(A):
    b: Column[StringType]


def foo(df: DataSet[A]) -> DataSet[A]:
    return transform_to_schema(
        df,
        B,
        {
            B.b: lit("hi"),
        },
    )


res = create_partially_filled_dataset(
    spark,
    A,
    {
        A.a: ["a", "b", "c"],
    },
).transform(foo)

```

--------------------------------

### Define DayTimeIntervalType

Source: https://typedspark.readthedocs.io/en/latest/api.html

Represents a time interval type, allowing for annotations with start and end fields like hours and seconds.

```python
class typedspark.DayTimeIntervalType
```

--------------------------------

### Initialize Databases Object

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Initializes the Databases object to load tables from the default catalog. This is useful when dealing with multiple catalogs or when the default catalog is slow to load.

```python
from typedspark import Databases

db = Databases()
```

--------------------------------

### DataSet.sample()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a new DataSet containing a random sample of this DataSet.

```APIDOC
## DataSet.sample()

### Description
Returns a new DataSet containing a random sample of this DataSet.

### Method
`DataSet.sample(withReplacement: bool, fraction: float, seed: int = None)`

### Parameters
- **withReplacement** (bool) - Required - Whether to sample with replacement.
- **fraction** (float) - Required - The fraction of rows to sample.
- **seed** (int) - Optional - The seed for the random number generator.

### Response
#### Success Response
- **DataSet** - A new DataSet containing the random sample.
```

--------------------------------

### Get schema column names excluding specified ones

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a list of all column names in a schema, excluding those provided in the `except_for` parameter.

```python
all_column_names_except_for(_except_for : List[str]) -> List[str]
```

--------------------------------

### Create a Partially Filled Dataset with a Dictionary

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates a DataSet with a specified Schema, filling data using a dictionary mapping columns to lists of values.

```python
class Person(Schema):
    name: Column[StringType]
    age: Column[LongType]
    job: Column[StringType]

df = create_partially_filled_dataset(
    spark,
    Person,
    {
        Person.name: ["John", "Jack", "Jane"],
        Person.age: [30, 40, 50],
    }
)
```

--------------------------------

### Create an Empty Dataset

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates a DataSet with a specified Schema and a given number of rows filled with None values.

```python
class Person(Schema):
    name: Column[StringType]
    age: Column[LongType]

df = create_empty_dataset(spark, Person)
```

--------------------------------

### Applying Generic Transformation to Different Datasets

Source: https://typedspark.readthedocs.io/en/latest/_sources/dataset_implements.ipynb.txt

Demonstrates applying the generic `birthday` transformation to `person` and `pet` datasets, and shows how it correctly raises an error for the `fruit` dataset which lacks an 'age' column.

```python
# returns a DataSet[Person]
happy_person = birthday(person)

# returns a DataSet[Pet]
happy_pet = birthday(pet)

try:
    # Raises a linting error:
    # Argument of type "DataSet[Fruit]" cannot be assigned to
    # parameter "df" of type "DataSetImplements[Age, T@birthday]"
    birthday(fruit)
except Exception as e:
    pass
```

--------------------------------

### DataSetImplements.join()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Joins this DataSetImplements with another based on a join expression.

```APIDOC
## DataSetImplements.join()

### Description
Joins this DataSetImplements with another based on a join expression.

### Method
`DataSetImplements.join(other: DataSetImplements, on: str = None, how: str = None)`

### Parameters
- **other** (DataSetImplements) - Required - The other DataSetImplements to join with.
- **on** (str) - Optional - The join expression.
- **how** (str) - Optional - The type of join (e.g., 'inner', 'left', 'right', 'full').

### Response
#### Success Response
- **DataSetImplements** - The resulting DataSetImplements after the join operation.
```

--------------------------------

### Load Table with Underscore Prefix (Renamed Attribute)

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Loads a table whose name starts with an underscore by accessing a renamed attribute (e.g., `u_person`) for auto-completion. The underlying table name remains unchanged.

```python
db = Database()

persons, Person = db.u_person()
```

--------------------------------

### Explicitly Map Ambiguous Columns in transform_to_schema

Source: https://typedspark.readthedocs.io/en/latest/_sources/column_ambiguity.ipynb.txt

This example resolves the `transform_to_schema` ambiguity by explicitly mapping the 'id' column from the 'person' schema to the 'PersonWithJob' schema. This ensures TypedSpark knows which 'id' column to use during the transformation.

```python
from typedspark import transform_to_schema


class PersonWithJob(Schema):
    id: Column[IntegerType]
    name: Column[StringType]
    age: Column[IntegerType]
    salary: Column[IntegerType]


(
    transform_to_schema(
        df_a.join(
            df_b,
            person.id == job.id,
        ),
        PersonWithJob,
        {
            PersonWithJob.id: person.id,
        }
    ).show()
)
```

--------------------------------

### DataSetImplements.checkpoint()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Marks the DataSetImplements for checkpointing.

```APIDOC
## DataSetImplements.checkpoint()

### Description
Marks the DataSetImplements for checkpointing.

### Method
`DataSetImplements.checkpoint()`

### Parameters
None

### Response
#### Success Response
- **DataSetImplements** - The DataSetImplements marked for checkpointing.
```

--------------------------------

### DataSet.hint()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Provides a hint to the optimizer.

```APIDOC
## DataSet.hint()

### Description
Provides a hint to the optimizer.

### Method
`DataSet.hint(name: str, value: list[str])`

### Parameters
- **name** (str) - Required - The name of the hint.
- **value** (list[str]) - Required - The value(s) for the hint.

### Response
#### Success Response
- **DataSet** - The DataSet with the applied hint.
```

--------------------------------

### sampleBy

Source: https://typedspark.readthedocs.io/en/latest/api.html

Samples data from the DataSet based on specified column fractions. Allows for setting a seed for reproducible sampling.

```APIDOC
## sampleBy

### Description
Samples data from the DataSet by specifying fractions for each value in a given column. A seed can be provided for deterministic results.

### Method
N/A (SDK method)

### Endpoint
N/A (SDK method)

### Parameters
- **_col_** (str) - Description: The column based on which to sample.
- **_fractions_** (Dict[str, float]) - Description: A dictionary where keys are column values and values are the desired sampling fractions.
- **_seed** (int | None) - Optional - Description: An integer seed for random number generation to ensure reproducibility.

### Request Example
N/A (SDK method)

### Response
#### Success Response
- **DataSet[_Schema]** - A new DataSet containing the sampled data.
```

--------------------------------

### DataSet.checkpoint()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Marks the DataSet for checkpointing.

```APIDOC
## DataSet.checkpoint()

### Description
Marks the DataSet for checkpointing.

### Method
`DataSet.checkpoint()`

### Parameters
None

### Response
#### Success Response
- **DataSet** - The DataSet marked for checkpointing.
```

--------------------------------

### Define Schemas and Create Empty Datasets

Source: https://typedspark.readthedocs.io/en/latest/dataset_implements.html

Defines three distinct schemas (Person, Pet, Fruit) with various column types and creates empty DataFrames for each using TypedSpark.

```python
from pyspark.sql.types import LongType, StringType
from typedspark import (
    Schema,
    Column,
    create_empty_dataset,
)


class Person(Schema):
    name: Column[StringType]
    age: Column[LongType]
    job: Column[StringType]


class Pet(Schema):
    name: Column[StringType]
    age: Column[LongType]
    type: Column[StringType]


class Fruit(Schema):
    type: Column[StringType]


person = create_empty_dataset(spark, Person)
pet = create_empty_dataset(spark, Pet)
fruit = create_empty_dataset(spark, Fruit)

```

--------------------------------

### Delta Live Table Definition with Schema Documentation

Source: https://typedspark.readthedocs.io/en/latest/_sources/documentation.ipynb.txt

Integrate typedspark schema documentation with Databricks Delta Live Tables. This allows the documentation to be displayed in the Databricks UI.

```python
@dlt.table(**Person.get_dlt_kwargs())
def table_definition() -> DataSet[Person]:
    # your table definition here
```

--------------------------------

### Database.str

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns the string representation of the Database.

```APIDOC
## Database.str

### Description
Returns the string representation of the Database.

### Method
`Database.str`

### Parameters
None

### Response
#### Success Response
- **str** (string) - The string representation of the Database.
```

--------------------------------

### Define a Schema with Documentation

Source: https://typedspark.readthedocs.io/en/latest/_sources/documentation.ipynb.txt

Define a schema for a person table, including metadata comments for each column. This sets up the structure and documentation for your data.

```python
from typing import Annotated
from pyspark.sql.types import DateType, StringType
from typedspark import Column, ColumnMeta, Schema


class Person(Schema):
    """Dimension table that contains information about a person."""

    person_id: Annotated[
        Column[StringType],
        ColumnMeta(comment="Unique person id"),
    ]
    gender: Annotated[Column[StringType], ColumnMeta(comment="Gender of the person")]
    birthdate: Annotated[Column[DateType], ColumnMeta(comment="Date of birth of the person")]
    job_id: Annotated[Column[StringType], ColumnMeta(comment="Id of the job")]
```

--------------------------------

### Load tables in a database

Source: https://typedspark.readthedocs.io/en/latest/api.html

Initializes a Database object to load all tables within a specified Spark database. Defaults to the 'default' database if none is provided.

```python
class typedspark.Database(_spark : SparkSession | None = None_, _db_name : str = 'default'_, _catalog_name : str | None = None_)
```

--------------------------------

### Print Schema with Documentation

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Prints the schema of a TypedSpark object, including any associated documentation. Useful for inspecting schema details.

```python
Person.print_schema(include_documentation=True)
```

--------------------------------

### DataSetImplements.repartition()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Repartitions the DataSetImplements to a specified number of partitions.

```APIDOC
## DataSetImplements.repartition()

### Description
Repartitions the DataSetImplements to a specified number of partitions.

### Method
`DataSetImplements.repartition(numPartitions: int)`

### Parameters
- **numPartitions** (int) - Required - The target number of partitions.

### Response
#### Success Response
- **DataSetImplements** - The repartitioned DataSetImplements.
```

--------------------------------

### create_empty_dataset()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates an empty DataSet with a specified schema.

```APIDOC
## create_empty_dataset()

### Description
Creates an empty DataSet with a specified schema.

### Method
`create_empty_dataset(schema: Schema)`

### Parameters
- **schema** (Schema) - Required - The schema for the empty DataSet.

### Response
#### Success Response
- **DataSet** - An empty DataSet with the given schema.
```

--------------------------------

### DataSetImplements.sampleBy()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a new DataSetImplements containing a stratified random sample of this DataSetImplements.

```APIDOC
## DataSetImplements.sampleBy()

### Description
Returns a new DataSetImplements containing a stratified random sample of this DataSetImplements.

### Method
`DataSetImplements.sampleBy(col: str, fractions: dict, seed: int = None)`

### Parameters
- **col** (str) - Required - The column to sample by.
- **fractions** (dict) - Required - A dictionary mapping values in `col` to the fraction of samples to draw.
- **seed** (int) - Optional - The seed for the random number generator.

### Response
#### Success Response
- **DataSetImplements** - A new DataSetImplements containing the stratified random sample.
```

--------------------------------

### create_partially_filled_dataset()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates a DataSet with partially filled data.

```APIDOC
## create_partially_filled_dataset()

### Description
Creates a DataSet with partially filled data.

### Method
`create_partially_filled_dataset(data: list[tuple], schema: Schema)`

### Parameters
- **data** (list[tuple]) - Required - The data to populate the DataSet.
- **schema** (Schema) - Required - The schema for the DataSet.

### Response
#### Success Response
- **DataSet** - A DataSet populated with the provided data.
```

--------------------------------

### Define Schema with Documentation

Source: https://typedspark.readthedocs.io/en/latest/documentation.html

Define a Spark SQL schema using TypedSpark, including comments for each column using ColumnMeta. This allows for structured documentation of your data.

```python
from typing import Annotated
from pyspark.sql.types import DateType, StringType
from typedspark import Column, ColumnMeta, Schema


class Person(Schema):
    """Dimension table that contains information about a person."""

    person_id: Annotated[
        Column[StringType],
        ColumnMeta(comment="Unique person id"),
    ]
    gender: Annotated[Column[StringType], ColumnMeta(comment="Gender of the person")]
    birthdate: Annotated[Column[DateType], ColumnMeta(comment="Date of birth of the person")]
    job_id: Annotated[Column[StringType], ColumnMeta(comment="Id of the job")]

```

--------------------------------

### Handling Duplicate Keys in `transform_to_schema` Transformations

Source: https://typedspark.readthedocs.io/en/latest/_sources/transforming_datasets.ipynb.txt

Demonstrates how providing duplicate keys in the `transformations` dictionary of `transform_to_schema` raises a `ValueError`. This highlights the requirement for unique column names as keys in the transformations.

```python
from typedspark import create_partially_filled_dataset

df = create_partially_filled_dataset(spark, Job, {Job.hourly_rate: [10, 20, 30]})

try:
    transform_to_schema(
        df,
        Job,
        {
            Job.hourly_rate: Job.hourly_rate + 3,
            Job.hourly_rate: Job.hourly_rate * 2,
        },
    )
except ValueError as e:
    print(e)
```

--------------------------------

### Sample DataSet by column fractions

Source: https://typedspark.readthedocs.io/en/latest/api.html

Samples rows from a DataSet based on specified fractions for a given column. A seed can be provided for reproducible sampling.

```python
sampleBy(_col_ , _fractions_ , _seed : int | None = None_) -> DataSet[_Implementation]
```

--------------------------------

### create_schema()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates a Schema object.

```APIDOC
## create_schema()

### Description
Creates a Schema object.

### Method
`create_schema(schema_definition: dict)`

### Parameters
- **schema_definition** (dict) - Required - A dictionary defining the schema.

### Response
#### Success Response
- **Schema** - The created Schema object.
```

--------------------------------

### Generate Schema Documentation

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Generates Python code for a typedspark Schema class based on the 'person_table' schema. This is useful for migrating existing tables to typedspark and generating documentation.

```python
from typing import Annotated

from pyspark.sql.types import LongType, StringType

from typedspark import Column, ColumnMeta, Schema


class PersonTable(Schema):

```

--------------------------------

### DataSetImplements.localCheckpoint()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Marks the DataSetImplements for local checkpointing.

```APIDOC
## DataSetImplements.localCheckpoint()

### Description
Marks the DataSetImplements for local checkpointing.

### Method
`DataSetImplements.localCheckpoint()`

### Parameters
None

### Response
#### Success Response
- **DataSetImplements** - The DataSetImplements marked for local checkpointing.
```

--------------------------------

### Load DataSet and Schema from Spark Catalog

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Loads a DataSet and its corresponding Schema from the default Spark catalog's 'person_table'. Autocomplete is available for table and column names.

```python
persons, Person = db.spark_catalog.default.person_table()
```

--------------------------------

### create_empty_dataset

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates an empty DataSet with a specified schema and number of rows, filled with None values.

```APIDOC
## create_empty_dataset

### Description
Creates a `DataSet` with `Schema` schema, containing `n_rows` rows, filled with `None` values.

### Parameters
- `_spark` (SparkSession) - The SparkSession instance.
- `_schema` (Type[T]) - The schema to use for the DataSet.
- `_n_rows` (int) - Optional - The number of rows to create. Defaults to 3.

### Returns
- DataSet[T] - A new DataSet with the specified schema and number of rows.
```

--------------------------------

### Create Empty Dataset from Schema

Source: https://typedspark.readthedocs.io/en/latest/_sources/create_empty_datasets.ipynb.txt

Generates an empty DataFrame with columns defined by the 'Person' schema. All fields will be null.

```python
from typedspark import Column, Schema, create_empty_dataset, create_partially_filled_dataset
from pyspark.sql.types import LongType, StringType


class Person(Schema):
    id: Column[LongType]
    name: Column[StringType]
    age: Column[LongType]


df_empty = create_empty_dataset(spark, Person)
df_empty.show()
```

--------------------------------

### Load Person Table using Databases

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Loads the 'person_table' dataset using the Databases object. It returns the dataset and its corresponding TypedSpark schema class.

```python
persons, Person = db.default.person_table()
```

--------------------------------

### load_table()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Loads a table into a DataSet.

```APIDOC
## load_table()

### Description
Loads a table into a DataSet.

### Method
`load_table(table_name: str, schema: Schema)`

### Parameters
- **table_name** (str) - Required - The name of the table to load.
- **schema** (Schema) - Required - The schema to apply to the loaded table.

### Response
#### Success Response
- **DataSet** - The DataSet loaded from the table.
```

--------------------------------

### Self-Join with Aliased Schema Registration

Source: https://typedspark.readthedocs.io/en/latest/_sources/column_ambiguity.ipynb.txt

Illustrates how to perform a self-join by registering the same schema to the dataset twice with different aliases. This is necessary when joining a dataset to itself to distinguish between the two instances of the schema.

```python
from typedspark import register_schema_to_dataset_with_alias

# Assuming spark and Person are defined elsewhere
# Example Person schema definition:
# from typedspark import Schema
# class Person(Schema):
#     id: int
#     name: str
#     age: int

# df = create_partially_filled_dataset(
#     spark,
#     Person,
#     {
#         Person.id: [1, 2, 3],
#         Person.name: ["Alice", "Bob", "Charlie"],
#         Person.age: [20, 30, 40],
#     },
# )

df_a, person_a = register_schema_to_dataset_with_alias(df, Person, alias="a")
df_b, person_b = register_schema_to_dataset_with_alias(df, Person, alias="b")

df_a.join(df_b, person_a.id == person_b.id).show()
```

--------------------------------

### Initialize Catalogs

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Initializes the Catalogs object from typedspark. This object is used to load DataSets and Schemas from the Spark catalog.

```python
from typedspark import Catalogs

db = Catalogs()
```

--------------------------------

### MetaSchema.get_dlt_kwargs()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Retrieves Delta Lake keyword arguments from the MetaSchema.

```APIDOC
## MetaSchema.get_dlt_kwargs()

### Description
Retrieves Delta Lake keyword arguments from the MetaSchema.

### Method
`MetaSchema.get_dlt_kwargs()`

### Parameters
None

### Response
#### Success Response
- **dlt_kwargs** (dict) - A dictionary of Delta Lake keyword arguments.
```

--------------------------------

### Create Partially Filled Dataset (Column-wise)

Source: https://typedspark.readthedocs.io/en/latest/_sources/create_empty_datasets.ipynb.txt

Creates a DataFrame with specified values for 'id' and 'name' columns, leaving 'age' as null. Useful for testing with specific column data.

```python
df_partially_filled = create_partially_filled_dataset(
    spark,
    Person,
    {
        Person.id: [1, 2, 3],
        Person.name: ["John", "Jane", "Jack"],
    },
)
df_partially_filled.show()
```

--------------------------------

### Delta Live Table Definition with TypedSpark Schema

Source: https://typedspark.readthedocs.io/en/latest/documentation.html

Integrate TypedSpark schemas with Delta Live Tables to automatically generate table documentation in the Databricks UI. This requires using the get_dlt_kwargs() method from the TypedSpark Schema.

```python
@dlt.table(**Person.get_dlt_kwargs())
def table_definition() -> DataSet[Person]:
    # your table definition here

```

--------------------------------

### Load Person Table using Database

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Loads the 'person_table' dataset using the Database object. It returns the dataset and its corresponding TypedSpark schema class.

```python
person, Person = db.person_table.load()
```

--------------------------------

### Corrected `transform_to_schema` with Unique Keys

Source: https://typedspark.readthedocs.io/en/latest/_sources/transforming_datasets.ipynb.txt

Shows the correct way to use `transform_to_schema` when applying multiple transformations to the same column by using separate lines for each transformation. This avoids the `ValueError` caused by duplicate keys.

```python
from typedspark import transform_to_schema

df = create_partially_filled_dataset(spark, Job, {Job.hourly_rate: [10, 20, 30]})

transform_to_schema(
    df,
    Job,
    {
        Job.hourly_rate: Job.hourly_rate + 3,
        Job.hourly_rate: Job.hourly_rate * 2,
    },
)
```

--------------------------------

### Concise DataFrame Transformation with `transform_to_schema`

Source: https://typedspark.readthedocs.io/en/latest/_sources/transforming_datasets.ipynb.txt

This snippet shows how to use the `transform_to_schema` function from `typedspark` for a more condensed transformation of a DataFrame to a target schema. It simplifies the process by handling casting and column selection implicitly.

```python
from typedspark import transform_to_schema

def get_plumbers(persons: DataSet[Person], jobs: DataSet[Job]) -> DataSet[PersonWithJob]:
    return transform_to_schema(
        jobs.filter(
            Job.function == "plumber",
        ).join(
            persons,
            Job.id == Person.job_id,
        ),
        PersonWithJob,
        {
            PersonWithJob.job_name: Job.function,
            PersonWithJob.rate: Job.hourly_rate,
        },
    )
```

--------------------------------

### Apply Typed Transformation

Source: https://typedspark.readthedocs.io/en/latest/_sources/advanced_linting_support.ipynb.txt

Demonstrates using the transform() method with a typed function (foo) that applies a schema transformation. The function transforms a DataSet of Schema A into a DataSet of Schema B.

```python
from typedspark import transform_to_schema
from pyspark.sql.functions import lit


class B(A):
    b: Column[StringType]


def foo(df: DataSet[A]) -> DataSet[A]:
    return transform_to_schema(
        df,
        B,
        {
            B.b: lit("hi"),
        },
    )


res = create_partially_filled_dataset(
    spark,
    A,
    {
        A.a: ["a", "b", "c"],
    },
).transform(foo)
```

--------------------------------

### DataSetImplements.limit()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a new DataSetImplements containing the first N rows.

```APIDOC
## DataSetImplements.limit()

### Description
Returns a new DataSetImplements containing the first N rows.

### Method
`DataSetImplements.limit(n: int)`

### Parameters
- **n** (int) - Required - The maximum number of rows to return.

### Response
#### Success Response
- **DataSetImplements** - A new DataSetImplements containing the first N rows.
```

--------------------------------

### create_partially_filled_dataset

Source: https://typedspark.readthedocs.io/en/latest/api.html

Creates a DataSet with a specified schema and partially filled data.

```APIDOC
## create_partially_filled_dataset

### Description
Creates a `DataSet` with `Schema` schema, where `data` can be defined in either of the following two ways:

### Parameters
- `_spark` (SparkSession) - The SparkSession instance.
- `_schema` (Type[T]) - The schema to use for the DataSet.
- `_data` (Dict[Column[Any], List[Any]] | List[Dict[Column[Any], Any]]) - The data to fill the DataSet with. Can be a dictionary mapping columns to lists of values, or a list of dictionaries representing rows.

### Returns
- DataSet[T] - A new DataSet with the specified schema and partially filled data.
```

--------------------------------

### DataSet.join()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Joins this DataSet with another based on a join expression.

```APIDOC
## DataSet.join()

### Description
Joins this DataSet with another based on a join expression.

### Method
`DataSet.join(other: DataSet, on: str = None, how: str = None)`

### Parameters
- **other** (DataSet) - Required - The other DataSet to join with.
- **on** (str) - Optional - The join expression.
- **how** (str) - Optional - The type of join (e.g., 'inner', 'left', 'right', 'full').

### Response
#### Success Response
- **DataSet** - The resulting DataSet after the join operation.
```

--------------------------------

### Create a StructType Column with Transformations

Source: https://typedspark.readthedocs.io/en/latest/api.html

Helps in creating new StructType columns by applying transformations to existing columns based on a defined schema.

```python
transform_to_schema(
    df,
    Output,
    {
        Output.values: structtype_column(
            Value,
            {
                Value.a: Input.a + 2,
                ...
            }
        )
    }
)
```

--------------------------------

### MetaSchema.print_schema()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Prints the schema of the MetaSchema.

```APIDOC
## MetaSchema.print_schema()

### Description
Prints the schema of the MetaSchema.

### Method
`MetaSchema.print_schema()`

### Parameters
None

### Response
None (prints schema to console).
```

--------------------------------

### Initialize Spark Session

Source: https://typedspark.readthedocs.io/en/latest/_sources/complex_datatypes.ipynb.txt

Initialize a SparkSession with disabled console progress and set the log level to ERROR. This is a prerequisite for generating datasets.

```python
from pyspark.sql import SparkSession

spark = SparkSession.Builder().config("spark.ui.showConsoleProgress", "false").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
```

--------------------------------

### DataSet.localCheckpoint()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Marks the DataSet for local checkpointing.

```APIDOC
## DataSet.localCheckpoint()

### Description
Marks the DataSet for local checkpointing.

### Method
`DataSet.localCheckpoint()`

### Parameters
None

### Response
#### Success Response
- **DataSet** - The DataSet marked for local checkpointing.
```

--------------------------------

### DataSetImplements.repartitionByRange()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Repartitions the DataSetImplements by range.

```APIDOC
## DataSetImplements.repartitionByRange()

### Description
Repartitions the DataSetImplements by range.

### Method
`DataSetImplements.repartitionByRange(numPartitions: int, *cols: str)`

### Parameters
- **numPartitions** (int) - Required - The target number of partitions.
- **cols** (str) - Required - The columns to partition by.

### Response
#### Success Response
- **DataSetImplements** - The repartitioned DataSetImplements.
```

--------------------------------

### Show DataFrame Contents

Source: https://typedspark.readthedocs.io/en/latest/_sources/loading_datasets_in_notebooks.ipynb.txt

Displays all rows and columns of the 'persons' DataFrame. This is useful for verifying the loaded data.

```python
persons.show()
```

--------------------------------

### Load a Table and Infer Schema

Source: https://typedspark.readthedocs.io/en/latest/api.html

Loads a Spark table into a DataSet and infers its Schema, returning both the DataSet and the Schema type.

```python
df, Person = load_table(spark, "path.to.table")
```

--------------------------------

### DataSetImplements.unionAll()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Returns a new DataSetImplements containing rows from both this DataSetImplements and another, including duplicates.

```APIDOC
## DataSetImplements.unionAll()

### Description
Returns a new DataSetImplements containing rows from both this DataSetImplements and another, including duplicates.

### Method
`DataSetImplements.unionAll(other: DataSetImplements)`

### Parameters
- **other** (DataSetImplements) - Required - The other DataSetImplements to union with.

### Response
#### Success Response
- **DataSetImplements** - A new DataSetImplements containing rows from both DataSetImplements, including duplicates.
```

--------------------------------

### MetaSchema.get_schema_definition_as_string()

Source: https://typedspark.readthedocs.io/en/latest/api.html

Retrieves the schema definition as a string.

```APIDOC
## MetaSchema.get_schema_definition_as_string()

### Description
Retrieves the schema definition as a string.

### Method
`MetaSchema.get_schema_definition_as_string()`

### Parameters
None

### Response
#### Success Response
- **schema_string** (str) - The schema definition as a string.
```
Tokens:
Raw