### Using PydanticJsonDataset for Pure Models (Python) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/README.md This Python example illustrates the direct usage of `PydanticJsonDataset` for 'pure', JSON-safe Pydantic models. It defines a `BaseModel`, instantiates it, saves it to an in-memory file using `fsspec`, and then reloads it, demonstrating the serialization and deserialization process. ```Python from pydantic import BaseModel # from pydantic.v1 import BaseModel # Pydantic V2 from pydantic_kedro import PydanticJsonDataset class MyPureModel(BaseModel): """Your custom Pydantic model with JSON-safe fields.""" x: int y: str obj = MyPureModel(x=1, y="why?") # Create an in-memory (temporary) file via `fsspec` and save it ds = PydanticJsonDataset("memory://temporary-file.json") ds.save(obj) # We can re-load it from the same file read_obj = ds.load() assert read_obj.x == 1 ``` -------------------------------- ### Saving and Loading Pydantic Models with Arbitrary Data (Pandas DataFrame) using pydantic-kedro (Python) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/standalone_usage.md This example illustrates how `pydantic-kedro` can handle Pydantic models containing arbitrary data types, specifically a Pandas DataFrame. It uses `ArbModel` and `ArbConfig` to map `pd.DataFrame` to `ParquetDataset` for serialization, demonstrating how to save and load a model with a DataFrame attribute while preserving data integrity. It requires `pydantic`, `pydantic-kedro`, `pandas`, and `kedro_datasets`. ```python from tempfile import TemporaryDirectory import pandas as pd from kedro_datasets.pandas.parquet_dataset import ParquetDataset from pydantic_kedro import ArbConfig, ArbModel, load_model, save_model # Arbitrary model class with a few useful defaults class _PdModel(ArbModel): """Pandas model, configured to use Parquet.""" class Config(ArbConfig): kedro_map = {pd.DataFrame: lambda x: ParquetDataset(filepath=x)} class MyModel(_PdModel): """My custom model.""" name: str data: pd.DataFrame df = pd.DataFrame({"x": [1, 2, 3], "y": ["a", "b", "c"]}) # We can use any fsspec URL, so we'll make a temporary folder with TemporaryDirectory() as tmpdir: save_model(MyModel(name="foo", data=df), f"{tmpdir}/my_model") obj = load_model(f"{tmpdir}/my_model") assert obj.data.equals(df) ``` -------------------------------- ### Folder and Zip Dataset Directory Structure Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/implementation_details.md Outlines the standard directory structure used by `PydanticFolderDataset` and `PydanticZipDataset`. It shows how data is organized within a `save_dir`, including a `meta.json` file for metadata and individual files/folders for model fields, prefixed with a dot. ```text save_dir |- meta.json |- .field1 |- .field2.0 | etc. ``` -------------------------------- ### JSON Dataset Structure for Pydantic Models Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/implementation_details.md Illustrates the structure of a JSON file generated by `PydanticJsonDataset`. It includes model fields and a special `"class"` field, which stores the full import path of the Pydantic model for self-description. This allows the dataset to automatically reconstruct the model upon loading. ```json { "foo": "bar", // the other fields from your model... "class": "your_module.Foo" } ``` -------------------------------- ### Saving and Loading Pure Pydantic Models with pydantic-kedro (Python) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/standalone_usage.md This snippet demonstrates the basic usage of `pydantic-kedro` to save and load a simple Pydantic `BaseModel`. It defines a `MyModel` with a string attribute, saves an instance to a temporary directory, and then loads it back, asserting the integrity of the saved data. It requires `pydantic` and `pydantic-kedro`. ```python from tempfile import TemporaryDirectory from pydantic import BaseModel # from pydantic.v1 import BaseModel # Pydantic V2 from pydantic_kedro import load_model, save_model class MyModel(BaseModel): """My custom model.""" name: str # We can use any fsspec URL, so we'll make a temporary folder with TemporaryDirectory() as tmpdir: save_model(MyModel(name="foo"), f"{tmpdir}/my_model") obj = load_model(f"{tmpdir}/my_model") assert obj.name == "foo" ``` -------------------------------- ### Standalone Pydantic Model Saving and Loading (Python) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/README.md This Python snippet demonstrates how to use `pydantic-kedro` as a standalone engine for saving and loading Pydantic models. It utilizes `save_model` and `load_model` functions with an `fsspec` URL pointing to a temporary directory, showcasing generic serialization capabilities outside of a Kedro catalog. ```Python from tempfile import TemporaryDirectory from pydantic.v1 import BaseModel from pydantic_kedro import load_model, save_model class MyModel(BaseModel): """My custom model.""" name: str # We can use any fsspec URL, so we'll make a temporary folder with TemporaryDirectory() as tmpdir: save_model(MyModel(name="foo"), f"{tmpdir}/my_model") obj = load_model(f"{tmpdir}/my_model") assert obj.name == "foo" ``` -------------------------------- ### Saving and Loading Pure Pydantic Models with PydanticJsonDataset (Python) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/index.md This Python snippet demonstrates standalone usage of `PydanticJsonDataset` for "pure" (JSON-serializable) Pydantic models. It defines `MyPureModel`, creates an instance, saves it to a temporary in-memory file using `fsspec`, and then loads it back, asserting the data integrity. This showcases direct serialization/deserialization outside of a Kedro pipeline. ```python from pydantic import BaseModel # from pydantic.v1 import BaseModel # Pydantic V2 from pydantic_kedro import PydanticJsonDataset class MyPureModel(BaseModel): """Your custom Pydantic model with JSON-safe fields.""" x: int y: str obj = MyPureModel(x=1, y="why?") # Create an in-memory (temporary) file via `fsspec` and save it ds = PydanticJsonDataset("memory://temporary-file.json") ds.save(obj) # We can re-load it from the same file read_obj = ds.load() assert read_obj.x == 1 ``` -------------------------------- ### Using Pydantic Models in Kedro Pipelines (Python) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/index.md This Python snippet shows how to define a Pydantic `BaseModel` and integrate it into a Kedro pipeline. It defines a simple `SomeModel` and a `my_func` that returns an instance of this model. The `node` function then connects `my_func`'s output to the `my_pydantic_model` defined in the catalog, enabling Kedro to manage the Pydantic model's lifecycle. ```python from pydantic import BaseModel # from pydantic.v1 import BaseModel # Pydantic V2 from kedro.pipeline import node class SomeModel(BaseModel): """My custom model.""" name: str def my_func(): return SomeModel(name="foo") my_func_node = node(my_func, outputs=["my_pydantic_model"]) # etc. ``` -------------------------------- ### Configuring PydanticAutoDataset in Kedro Catalog (YAML) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/README.md This YAML snippet demonstrates how to configure a `PydanticAutoDataset` within a Kedro `catalog.yml` file. It allows Kedro to automatically handle the serialization and deserialization of Pydantic models, specifying the dataset type and the file path for storage. ```YAML # conf/base/catalog.yml my_pydantic_model: type: pydantic_kedro.PydanticAutoDataset filepath: folder/my_model ``` -------------------------------- ### Configuring PydanticAutoDataset in Kedro Catalog (YAML) Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/index.md This YAML snippet demonstrates how to configure a `PydanticAutoDataset` in Kedro's `catalog.yml` file. It maps a dataset name (`my_pydantic_model`) to the `pydantic_kedro.PydanticAutoDataset` type and specifies the `filepath` where the model will be saved. This allows Kedro to automatically handle serialization and deserialization of Pydantic models within pipelines. ```yaml # conf/base/catalog.yml my_pydantic_model: type: pydantic_kedro.PydanticAutoDataset filepath: folder/my_model ``` -------------------------------- ### Defining Pydantic Models with Arbitrary Types using ArbModel in Python Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/arbitrary_types.md This snippet illustrates the use of `pydantic_kedro.ArbModel` as a shorthand for defining Pydantic models that allow arbitrary types. `ArbModel` automatically sets `arbitrary_types_allowed = True` in the model's configuration, simplifying the definition of models with non-standard fields. ```Python from pydantic_kedro import ArbModel class MyArbitraryModel(ArbModel): """Your custom Pydantic model with JSON-unsafe fields.""" x: int foo: Foo ``` -------------------------------- ### Customizing Type Serialization with kedro_map in Pydantic-Kedro for Pandas DataFrames Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/arbitrary_types.md This snippet demonstrates how to explicitly define a dataset for a specific type (e.g., `pandas.DataFrame`) using the `kedro_map` configuration within a Pydantic model. By mapping `pd.DataFrame` to `ParquetDataset`, `pydantic-kedro` serializes DataFrames into a more robust and future-proof Apache Parquet format instead of the default Pickle, ensuring better compatibility and readability. ```Python import pandas as pd from kedro_datasets.pandas import ParquetDataset from pydantic import validator # from pydantic.v1 import validator # Pydantic V2 from pydantic_kedro import ArbModel, PydanticZipDataset class MyPandasModel(ArbModel): """Model that saves a dataframe, along with some other data.""" class Config: kedro_map = {pd.DataFrame: ParquetDataset} val: int df: pd.DataFrame @validator('df') def _check_dataframe(cls, v: pd.DataFrame) -> pd.DataFrame: """Ensure the dataframe is valid.""" assert len(v) > 0 return v dfx = pd.DataFrame([[1, 2, 3]], columns=["a", "b", "c"]) m1 = MyPandasModel(df=dfx, val=1) ds = PydanticZipDataset(f"memory://my_model.zip") ds.save(m1) m2 = ds.load() assert m2.df.equals(dfx) ``` -------------------------------- ### Serializing Pydantic Models with Arbitrary Types using PydanticZipDataset in Python Source: https://github.com/nowanilfideme/pydantic-kedro/blob/main/docs/arbitrary_types.md This snippet demonstrates how to define a Pydantic model with an arbitrary, non-JSON-serializable type (`Foo`) and serialize it using `PydanticZipDataset`. It shows that direct JSON serialization fails, but `PydanticZipDataset` successfully saves and loads the object, preserving its custom type. The `arbitrary_types_allowed = True` configuration is crucial for this functionality. ```Python from tempfile import TemporaryDirectory from pydantic import BaseModel # from pydantic.v1 import BaseModel # Pydantic V2 from pydantic_kedro import PydanticZipDataset class Foo(object): """My custom class. NOTE: this is not a Pydantic model!""" def __init__(self, foo): self.foo = foo class MyArbitraryModel(BaseModel): """Your custom Pydantic model with JSON-unsafe fields.""" x: int foo: Foo class Config: """Configuration for Pydantic V1.""" # Let's pretend it would be difficult to add a json encoder for Foo arbitrary_types_allowed = True obj = MyArbitraryModel(x=1, foo=Foo("foofoo")) # This object is not JSON-serializable try: obj.json() except TypeError as err: print(err) # Object of type 'Foo' is not JSON serializable # We can, however, with TemporaryDirectory() as tmpdir: # Create an on-disk (temporary) file via `fsspec` and save it ds = PydanticZipDataset(f"{tmpdir}/arb.zip") ds.save(obj) # We can re-load it from the same file read_obj = ds.load() assert read_obj.foo.foo == "foofoo" ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.