### DataVar Initialization Example Source: https://github.com/unionai-oss/pandera/blob/main/specs/xarray.md Example of initializing a DataVar with various parameters. ```APIDOC ```python from pandera.api.pandera.schema_components import DataVar # Example DataVar definition temperature_var = DataVar( dtype="float64", dims=("time", "lat", "lon"), required=True, alias="temp", checks=[ # Add checks here, e.g., pandera.Check.greater_than(0) ], description="Temperature data variable" ) # Example of an optional variable with a default pressure_var = DataVar( dtype="float32", dims=("time", "lat", "lon"), required=False, default=0.0, description="Pressure data variable (optional)" ) # Example using regex for matching variable names wind_vars = DataVar( regex=True, dtype="float32", dims=("time", "lat", "lon"), description="Wind components (e.g., u_wind, v_wind)" ) ``` ``` -------------------------------- ### Install Pandera strategies Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/xarray_guide/hypothesis_strategies.md Install the necessary dependencies to use Hypothesis strategies with Pandera. ```bash pip install 'pandera[strategies]' ``` -------------------------------- ### Install Pandera Source: https://github.com/unionai-oss/pandera/blob/main/README.md Installation commands for Pandera using various package managers. ```bash pip install 'pandera[pandas]' ``` ```bash uv pip install 'pandera[pandas]' ``` ```bash conda install -c conda-forge pandera-pandas ``` -------------------------------- ### Install Pandera with Narwhals support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/narwhals_backend.md Install the required extras for Narwhals and the specific backend being used. ```bash pip install 'pandera[narwhals,polars]' # Polars pip install 'pandera[narwhals,ibis]' # Ibis pip install 'pandera[narwhals,pyspark]' # PySpark SQL ``` -------------------------------- ### Install Pandera extras Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/index.md Install additional functionality and library support using pip or conda. ```bash pip install 'pandera[hypotheses]' # hypothesis checks pip install 'pandera[io]' # yaml/script schema io utilities pip install 'pandera[frictionless]' # frictionless schema serialization/deserialization pip install 'pandera[strategies]' # data synthesis strategies pip install 'pandera[mypy]' # enable static type-linting of pandas pip install 'pandera[fastapi]' # fastapi integration pip install 'pandera[dask]' # validate dask dataframes pip install 'pandera[pyspark]' # validate pyspark dataframes pip install 'pandera[modin]' # validate modin dataframes pip install 'pandera[modin-ray]' # validate modin dataframes with ray pip install 'pandera[modin-dask]' # validate modin dataframes with dask pip install 'pandera[geopandas]' # validate geopandas geodataframes pip install 'pandera[polars]' # validate polars dataframes pip install 'pandera[ibis]' # validate ibis tables pip install 'pandera[xarray]' # validate xarray data structures pip install 'pandera[narwhals]' # use the Narwhals-powered backend ``` ```bash conda install -c conda-forge pandera-hypotheses # hypothesis checks conda install -c conda-forge pandera-io # yaml/script schema io utilities conda install -c conda-forge pandera-frictionless # frictionless schema serialization/deserialization conda install -c conda-forge pandera-strategies # data synthesis strategies conda install -c conda-forge pandera-mypy # enable static type-linting of pandas conda install -c conda-forge pandera-fastapi # fastapi integration conda install -c conda-forge pandera-dask # validate dask dataframes conda install -c conda-forge pandera-pyspark # validate pyspark dataframes conda install -c conda-forge pandera-modin # validate modin dataframes conda install -c conda-forge pandera-modin-ray # validate modin dataframes with ray conda install -c conda-forge pandera-modin-dask # validate modin dataframes with dask conda install -c conda-forge pandera-geopandas # validate geopandas geodataframes conda install -c conda-forge pandera-polars # validate polars dataframes conda install -c conda-forge pandera-ibis # validate ibis tables conda install -c conda-forge pandera-narwhals # use the Narwhals-powered backend ``` -------------------------------- ### Install Pandera Mypy support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/mypy_integration.md Install the necessary extra dependencies to enable Mypy integration. ```bash pip install pandera[mypy] ``` -------------------------------- ### Install Fugue with Spark and Dask Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/fugue.md Installs the Fugue library with support for Spark and Dask. This command ensures that PySpark is also installed when the 'spark' extra is selected. ```bash pip install 'fugue[spark]' ``` -------------------------------- ### Install Pandera with Polars support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/polars.md Install the necessary dependencies to use Pandera with Polars. ```bash pip install 'pandera[polars]' ``` -------------------------------- ### Install and enable Narwhals-powered backend Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/ibis.md Install the Narwhals extra and enable the backend via environment variable or configuration function. ```bash pip install 'pandera[ibis,narwhals]' export PANDERA_USE_NARWHALS_BACKEND=True ``` ```python import pandera.ibis as pa pa.set_config(use_narwhals_backend=True) ``` -------------------------------- ### Install Pandera with Ibis support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/ibis.md Install the necessary packages to use Pandera with Ibis and a specific backend. ```bash pip install 'pandera[ibis]' 'ibis-framework[duckdb]' ``` -------------------------------- ### Install Pandera with Modin Support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/modin.md Installs pandera with the necessary extras for Modin integration. This includes options for installing with both Dask and Ray backends, or specific backends. ```bash pip install 'pandera[modin]' # installs both ray and dask backends pip install 'pandera[modin-ray]' # only ray backend pip install 'pandera[modin-dask]' # only dask backend ``` -------------------------------- ### Install Pandera with Dask support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/dask.md Command to install the necessary dependencies for using Pandera with Dask. ```bash pip install 'pandera[dask]' ``` -------------------------------- ### Install Narwhals backend for PySpark Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/pyspark_sql.md Install the required extras and set the environment variable to enable the lazy Narwhals backend. ```bash pip install 'pandera[pyspark,narwhals]' export PANDERA_USE_NARWHALS_BACKEND=True ``` -------------------------------- ### Install Pandera Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/notebooks/try_pandera.ipynb Installs the Pandera library using pip. This is a prerequisite for using Pandera in your Python environment. ```python !pip install pandera ``` -------------------------------- ### Install Pandera with Xarray Support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/xarray_guide/index.md Install pandera with the xarray extra to enable xarray data validation. ```bash pip install 'pandera[xarray]' ``` -------------------------------- ### Install Pandera with pandas support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/index.md Install the core Pandera package with pandas integration using common package managers. ```bash pip install 'pandera[pandas]' ``` ```bash uv pip install 'pandera[pandas]' ``` ```bash conda install -c conda-forge pandera-pandas ``` -------------------------------- ### Install Pandera with GeoPandas support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/geopandas.md Install the necessary dependencies to enable GeoPandas validation features. ```bash pip install 'pandera[geopandas]' ``` -------------------------------- ### Install pandera with PySpark support Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/pyspark.md Install the pandera library with the necessary extras for PySpark integration. This command ensures all dependencies for using pandera with PySpark DataFrames are met. ```bash pip install 'pandera[pyspark]' ``` -------------------------------- ### Create a DataFrame for Validation Source: https://github.com/unionai-oss/pandera/blob/main/README.md Initial setup of a pandas DataFrame to be validated by Pandera. ```python import pandas as pd import pandera.pandas as pa # data to validate df = pd.DataFrame({ "column1": [1, 2, 3], "column2": [1.1, 1.2, 1.3], "column3": ["a", "b", "c"], }) ``` -------------------------------- ### Strategies and Examples from DataFrame Models Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/data_synthesis_strategies.md Generate examples and strategies from DataFrame models. ```python from pandera.typing import Series, DataFrame class InSchema(pa.DataFrameModel): column1: Series[int] = pa.Field(eq=10) column2: Series[float] = pa.Field(eq=0.25) column3: Series[str] = pa.Field(eq="foo") class OutSchema(InSchema): column4: Series[float] @pa.check_types def processing_fn(df: DataFrame[InSchema]) -> DataFrame[OutSchema]: return df.assign(column4=df.column1 * df.column2) @hypothesis.given(InSchema.strategy(size=5)) def test_processing_fn(dataframe): processing_fn(dataframe) ``` -------------------------------- ### Importing Frictionless Data Schema Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/frictionless.md Converts a Frictionless Data schema into a Pandera schema. Requires the 'io' extension to be installed. ```python .. autofunction:: pandera.io.pandas_io.from_frictionless_schema ``` -------------------------------- ### Enable Narwhals backend via environment variable Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/narwhals_backend.md Set the environment variable before starting the Python process to enable the backend. ```bash export PANDERA_USE_NARWHALS_BACKEND=True python your_script.py ``` -------------------------------- ### Basic Usage Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/data_synthesis_strategies.md Generate example data from a pandera schema. ```python import pandera.pandas as pa schema = pa.DataFrameSchema( { "column1": pa.Column(int, pa.Check.eq(10)), "column2": pa.Column(float, pa.Check.eq(0.25)), "column3": pa.Column(str, pa.Check.eq("foo")), } ) schema.example(size=3) ``` -------------------------------- ### Implement Path-Based DataTreeSchema Source: https://github.com/unionai-oss/pandera/blob/main/specs/xarray.md Example of defining a schema for a hierarchical DataTree using path-based child definitions. ```python schema = pa.DataTreeSchema( dataset=pa.DatasetSchema(attrs={"conventions": "CF-1.8"}), children={ "surface": pa.DatasetSchema( data_vars={ "temperature": pa.DataVar( dtype=np.float64, dims=("time", "lat", "lon"), ), }, ), "surface/diagnostics": pa.DatasetSchema( data_vars={"rmse": pa.DataVar(dtype=np.float64)}, ), }, strict=True, ) ``` -------------------------------- ### Stacked constraints example Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md Demonstrates how multiple user-defined constraints can be stacked on the same column, showing the intersection of bounds and the effect of `post_merge_hook`. ```python pa.Column(int, checks=[ Check.gt(0), # min_value=0, excl Check.lt(100_000), # max_value=100k, excl Check.divisible_by(divisor=5), # residual + (v2: isin) Check.percentile_clipped(ref_low=50, ref_high=90_000), # min=50, max=90k ]) ``` -------------------------------- ### Registering and Using Statistic DataFrame Checks Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/extensions.md This example shows how to register custom check methods for DataFrame-level validation and then use them within a DataFrameModel's Config. ```python import pandera.pandas as pa import pandera.extensions as extensions import numpy as np import pandas as pd @extensions.register_check_method() def is_small(df): return sum(df.shape) < 1000 @extensions.register_check_method(statistics=["fraction"]) def total_missing_fraction_less_than(df, *, fraction: float): return (1 - df.count().sum().item() / df.apply(len).sum().item()) < fraction @extensions.register_check_method(statistics=["col_a", "col_b"]) def col_mean_a_greater_than_b(df, *, col_a: str, col_b: str): return df[col_a].mean() > df[col_b].mean() from pandera.typing import Series class Schema(pa.DataFrameModel): col1: Series[float] = pa.Field(nullable=True, ignore_na=False) col2: Series[float] = pa.Field(nullable=True, ignore_na=False) class Config: is_small = () total_missing_fraction_less_than = 0.6 col_mean_a_greater_than_b = {"col_a": "col2", "col_b": "col1"} data = pd.DataFrame({ "col1": [float('nan')] * 3 + [0.5, 0.3, 0.1], "col2": np.arange(6.), }) Schema.validate(data) ``` -------------------------------- ### Initialize Development Environment Source: https://github.com/unionai-oss/pandera/blob/main/AGENTS.md Commands to set up the local development environment using uv. ```bash # Install uv and sync all extras make setup # macOS (uses polars-lts-cpu) make setup-macos ``` -------------------------------- ### Build Documentation Source: https://github.com/unionai-oss/pandera/blob/main/AGENTS.md Commands to generate project documentation using Sphinx. ```bash # Full build with doctests (cleans first) make docs # Quick build (no clean, no -W flag) make quick-docs # Via nox nox -db uv -s docs ``` -------------------------------- ### Publish and Preview ASV Benchmark Results Source: https://github.com/unionai-oss/pandera/blob/main/asv_bench/README.md Builds the HTML documentation for the benchmark results and provides a local preview. This is a crucial step before publishing the results to a website or repository. It uses the specified ASV configuration file. ```bash asv publish --config asv_bench/asv.conf.json asv preview --config asv_bench/asv.conf.json ``` -------------------------------- ### Enable Narwhals backend via configuration Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/index.md Use the pandera configuration API to enable the Narwhals backend at runtime. ```python import pandera pandera.set_config(use_narwhals_backend=True) ``` -------------------------------- ### Numeric Strategy Example Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md Microbenchmark example for generating rows with a combination of numeric checks. ```python Check.gt(0) & Check.lt(1e10) & Check.notin([-100,-10,0]) ``` -------------------------------- ### Enable Narwhals backend via configuration Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/pyspark_sql.md Programmatically enable the Narwhals backend using the Pandera configuration API. ```python import pandera.pyspark as pa pa.set_config(use_narwhals_backend=True) ``` -------------------------------- ### ConstraintConflictError Message Example Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md Example of the error message format when a ConstraintConflictError is caught and re-raised as SchemaDefinitionError. ```text Cannot construct a data-generation strategy for column 'x' with checks [gt(10), lt(5)]: constraints are jointly unsatisfiable (min_value=10 > max_value=5). ``` -------------------------------- ### SchemaDefinitionError example Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md An example of a SchemaDefinitionError raised when checks are jointly unsatisfiable, indicating a conflict between constraints. ```python SchemaDefinitionError: Cannot construct a data-generation strategy for column 'amount_cents' with checks [gt(0), lt(100000), divisible_by(divisor=5), percentile_clipped(ref_low=200000, ref_high=300000)]: constraints are jointly unsatisfiable (min_value=200000 > max_value=100000; contributed by 'percentile_clipped' and 'lt'). ``` -------------------------------- ### ASV Reporting Example Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md Example markdown table generated by the ASV reporting CLI for comparing benchmarks. ```markdown | Scenario | Metric | Baseline | New | Δ | Pass? | |---|---|---|---|---|---| | AggregatedNumericChecks(n=8) | time_example_size_100 | 4.20 s | 0.62 s | 6.8× | ✓ | | AggregatedNumericChecks(n=8) | filter_node_count | 10 | 1 | -9 | ✓ | | StringRegexMerging | time_example_size_100 | 1.10 s | 0.18 s | 6.1× | ✓ | | StringRegexMerging | regex_filter_node_count | 3 | 0 | -3 | ✓ | | ... | | | | | | ``` -------------------------------- ### Run Linting and Formatting Hooks Source: https://github.com/unionai-oss/pandera/blob/main/AGENTS.md Manually trigger pre-commit hooks to enforce code quality standards. ```bash prek run --all-files ``` -------------------------------- ### Example schema with custom divisible_by check Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md An example of how to define a pandera schema with a custom 'divisible_by_5' check, demonstrating the use of Check.gt, Check.lt, and a lambda function for the divisibility constraint. ```python import pandera.pandas as pa from pandera.api.checks import Check schema = pa.DataFrameSchema({ "amount_cents": pa.Column(int, checks=[ Check.gt(0), Check.lt(100_000), Check( lambda s: (s % 5) == 0, name="divisible_by_5", error="must be a multiple of 5 cents", ), ]), }) schema.example(size=100) # slow; can hit Unsatisfiable ``` -------------------------------- ### Strategy for in_between check Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/extensions.md The strategy for the is_between function example. ```python def in_between_strategy( pandera_dtype: pa.DataType, strategy: Optional[st.SearchStrategy] = None, *, min_value, max_value ): if strategy is None: return st.pandas_dtype_strategy( pandera_dtype, min_value=min_value, max_value=max_value, exclude_min=False, exclude_max=False, ) return strategy.filter(lambda x: min_value <= x <= max_value) ``` -------------------------------- ### Enable Narwhals backend programmatically Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/narwhals_backend.md Use set_config to toggle the backend within a running Python process. ```python import pandera pandera.set_config(use_narwhals_backend=True) import pandera.polars as pa ``` -------------------------------- ### Inline Custom Checks Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/extensions.md Examples of inline custom checks for pandera. ```python import pandera.pandas as pa # checks elements in a column/dataframe element_wise_check = pa.Check(lambda x: x < 0, element_wise=True) # applies the check function to a dataframe/series vectorized_check = pa.Check(lambda series_or_df: series_or_df < 0) ``` -------------------------------- ### Convert model to schema and validate Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/xarray_guide/data_models.md Demonstrates converting a model to a schema object and performing validation. ```python schema = Temperature.to_schema() print(type(schema)) Temperature.validate(da) ``` -------------------------------- ### Using type-checking decorators Source: https://github.com/unionai-oss/pandera/blob/main/specs/xarray.md Example of using the @check_types decorator with xarray type annotations. ```python from pandera.typing.xarray import DataArray, Dataset, Coordinate @pa.check_types def process( ds: Dataset[ClimateData], ) -> Dataset[ClimateData]: ... ``` -------------------------------- ### Update Pandera Imports Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/index.md Demonstrates the transition from the deprecated top-level module to the recommended pandera.pandas module. ```python import pandera as pa schema = pa.DataFrameSchema({"col": pa.Column(str)}) ``` ```python import pandera.pandas as pa ``` -------------------------------- ### Execute Test Suites Source: https://github.com/unionai-oss/pandera/blob/main/AGENTS.md Commands for running backend-specific tests, coverage reports, and parameterized tests via nox. ```bash # Run core + pandas tests pytest tests/core tests/pandas # Run a specific backend's tests pytest tests/polars/ pytest tests/pyspark/ pytest tests/ibis/ # Run all tests with coverage pytest --cov=pandera --cov-report=term-missing tests/ # Run via nox (parameterized across Python/pandas/pydantic/polars versions) nox -db uv -s tests # Run a specific nox session nox -db uv -s "tests(extra='polars', pandas=None, pydantic=None, polars='1.33.1')" ``` -------------------------------- ### Registering Custom Check Methods Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/extensions.md Example of registering a custom check method 'is_between' to pandera. ```python import pandera.pandas as pa import pandera.extensions as extensions import pandas as pd @extensions.register_check_method(statistics=["min_value", "max_value"]) def is_between(pandas_obj, *, min_value, max_value): return (min_value <= pandas_obj) & (pandas_obj <= max_value) schema = pa.DataFrameSchema({ "col": pa.Column(int, pa.Check.is_between(min_value=1, max_value=10)) }) data = pd.DataFrame({"col": [1, 5, 10]}) schema.validate(data) ``` -------------------------------- ### Implement Custom Converters with Callables Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/data_format_conversion.md Illustrates how to use custom functions for serialization by passing callables to the 'to_format' and 'to_format_buffer' configuration options. ```python import tempfile def custom_to_pickle(data, *args, **kwargs): return data.to_pickle(*args, **kwargs) def custom_to_pickle_buffer(): return tempfile.NamedTemporaryFile() class OutSchemaPickleCallable(OutSchema): class Config: to_format = custom_to_pickle to_format_buffer = custom_to_pickle_buffer ``` -------------------------------- ### Validate DataArray attributes via regex Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/xarray_guide/data_array_schema.md Use regex patterns starting with ^ to validate attribute values. ```python schema = pa.DataArraySchema( attrs={"units": "^(K|degC|degF)$"}, ) da_units = xr.DataArray( np.ones(3), dims="x", attrs={"units": "K"}, ) schema.validate(da_units) ``` ```python da_bad_units = xr.DataArray( np.ones(3), dims="x", attrs={"units": "meters"}, ) try: schema.validate(da_bad_units) except pa.errors.SchemaError as exc: print(exc) ``` -------------------------------- ### Configuring Data Format Conversion Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/data_format_conversion.md How to use the Config class to define input and output formats for Pandera schemas. ```APIDOC ## Data Format Configuration ### Description Configure a schema to automatically handle data serialization (to_format) and deserialization (from_format) when used with the @check_types decorator. ### Parameters #### Config Options - **from_format** (str or callable) - Optional - Specifies the format or function to read input data (e.g., 'parquet', 'csv', or custom callable). - **to_format** (str or callable) - Optional - Specifies the format or function to write output data (e.g., 'dict', 'json', or custom callable). - **to_format_kwargs** (dict) - Optional - Keyword arguments passed to the pandas serialization method. - **to_format_buffer** (callable) - Optional - A function that returns a buffer or file handle for writing data. ### Request Example class InSchemaParquet(pa.DataFrameModel): class Config: from_format = "parquet" class OutSchemaDict(pa.DataFrameModel): class Config: to_format = "dict" to_format_kwargs = {"orient": "records"} ### Supported Formats - dict - csv - json - feather - parquet - pickle - Custom Callable ``` -------------------------------- ### Enable Narwhals backend Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/configuration.md Enable the Narwhals-powered backend using either environment variables or the Python configuration API. ```bash export PANDERA_USE_NARWHALS_BACKEND=True ``` ```python import pandera pandera.set_config(use_narwhals_backend=True) ``` -------------------------------- ### Using DataTree with @check_types Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/xarray_guide/data_tree.md Use DataTree[Model] from pandera.typing.xarray with the @check_types decorator. Ensure pandera and xarray are installed. ```python from pandera.typing.xarray import DataTree @pa.check_types def process_tree(tree: DataTree[ClimateTree]) -> DataTree[ClimateTree]: return tree process_tree(dt) ``` -------------------------------- ### GET /pandera/strategies/pandas_strategies Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/reference/strategies.rst Access the pandas-specific data synthesis strategies for generating synthetic dataframes based on schema definitions. ```APIDOC ## GET /pandera/strategies/pandas_strategies ### Description Retrieves the collection of strategies used for generating synthetic pandas objects. These strategies are typically used in conjunction with schema definitions to create mock data for testing pipelines. ### Method GET ### Endpoint /pandera/strategies/pandas_strategies ### Parameters None ### Request Example GET /pandera/strategies/pandas_strategies ### Response #### Success Response (200) - **strategies** (object) - A dictionary mapping data types or schema components to their respective synthesis strategies. #### Response Example { "strategies": { "int64": "", "float64": "", "string": "" } } ``` -------------------------------- ### Lazy Backend Registration Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/narwhals_backend.md Demonstrates setting the Narwhals backend configuration before schema construction to trigger lazy registration. ```python import pandera.polars as pa # CONFIG.use_narwhals_backend is read here — not at import time above pa.config.set_config(use_narwhals_backend=True) schema = pa.DataFrameSchema({"name": pa.Column(str)}) # narwhals backends registered schema.validate(df) ``` -------------------------------- ### Report DataTree Validation Errors Source: https://github.com/unionai-oss/pandera/blob/main/specs/xarray.md Example of a SchemaError indicating the specific path to a failing node within a DataTree structure. ```python SchemaError( "DataTreeSchema failed at path '/surface/diagnostics': " "data variable 'rmse' not found in Dataset" ) ``` -------------------------------- ### Run All Benchmarks with ASV Source: https://github.com/unionai-oss/pandera/blob/main/asv_bench/README.md Executes all defined benchmarks using Airspeed Velocity. This command requires both the pandera and pandera-asv-logs repositories to be checked out in the same parent directory. It utilizes a specific configuration file for the benchmark run. ```bash asv run ALL --config asv_bench/asv.conf.json ``` -------------------------------- ### DatasetSchema Initialization Source: https://github.com/unionai-oss/pandera/blob/main/specs/xarray.md Demonstrates the initialization of a DatasetSchema with various parameters to define validation rules for an xarray.Dataset. ```APIDOC ## DatasetSchema Initialization ### Description Initializes a `DatasetSchema` object to define validation rules for an `xr.Dataset`. ### Method `__init__` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **data_vars** (`dict[str, DataVar | None] | None`) - Required/Optional - A dictionary mapping data variable names to their `DataVar` specifications. Keys can also be regex patterns. `None` indicates required presence without value-level validation. - **coords** (`dict[str, Coordinate] | list[str] | None`) - Required/Optional - Dataset-level coordinate schemas or a list of coordinate names to check for existence. - **dims** (`tuple[str, ...] | None`) - Required/Optional - Expected dataset-level dimension names. If `ordered_dims` is `False`, only the set of names is checked. - **ordered_dims** (`bool`) - Required/Optional - If `True` (default), checks dimension names in order. If `False`, only checks for the presence of the set of names. - **sizes** (`dict[str, int | None] | None`) - Required/Optional - Expected dataset-level dimension sizes as a name-to-length mapping. - **attrs** (`dict[str, Any] | type[BaseModel] | None`) - Required/Optional - Expected dataset-level attributes. Can be a dictionary with literal values, regex patterns (strings starting with `^`), callables, or a pydantic `BaseModel` class. - **checks** (`Check | list[Check] | None`) - Required/Optional - Dataset-wide checks that receive the full `xr.Dataset`. - **parsers** (`Parser | list[Parser] | None`) - Required/Optional - Parsers to apply to the dataset. - **strict** (`bool | "filter"`) - Required/Optional - If `True`, fail on unexpected data variables. If `"filter"`, drop them. - **strict_coords** (`bool`) - Required/Optional - If `True`, fail on unexpected dataset-level coordinate keys. - **strict_attrs** (`bool`) - Required/Optional - If `True`, fail on unexpected attributes. - **name** (`str | None`) - Required/Optional - The name of the schema. - **title** (`str | None`) - Required/Optional - The title of the schema. - **description** (`str | None`) - Required/Optional - The description of the schema. - **metadata** (`dict | None`) - Required/Optional - Additional metadata for the schema. ### Request Example ```python import pandera as pa import xarray as xr import numpy as np schema = pa.DatasetSchema( data_vars={ "temperature": pa.DataVar( dtype=np.float64, dims=("time", "lat", "lon"), checks=pa.Check.in_range(150, 350), ), "precipitation": pa.DataVar( dtype=np.float64, dims=("time", "lat", "lon"), checks=pa.Check.ge(0), ), "uncertainty": pa.DataVar( dtype=np.float64, dims=("time", "lat", "lon"), required=False, ), }, coords={ "time": pa.Coordinate(dtype="datetime64[ns]"), "lat": pa.Coordinate(dtype=np.float64), "lon": pa.Coordinate(dtype=np.float64), }, dims=("time", "lat", "lon"), sizes={ "time": 100, "lat": 10, "lon": 20, }, attrs={ "source": "met_office", "history": lambda v: isinstance(v, str) and v.startswith("2023-") }, checks=pa.Check(lambda ds: ds.dims["time"] == ds.coords["time"].size, element_wise=False), strict=True, strict_coords=True, strict_attrs=True, ) ``` ### Response None (This is an initializer) ``` -------------------------------- ### Compiled strategy example Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md The adapter above compiles to a strategy that uses `st.sampled_from` for efficiency. ```python strat = pandas_dtype_strategy( pa.Int(), st.sampled_from(sorted(range(5, 100_000, 5))), ) # zero filters; every draw is valid by construction. ``` -------------------------------- ### Configure Mypy plugin Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/mypy_integration.md Enable the Pandera plugin in your Mypy configuration file. ```ini [mypy] plugins = pandera.mypy ``` -------------------------------- ### Validate xarray Dataset with Coordinate Schema Source: https://github.com/unionai-oss/pandera/blob/main/specs/xarray.md Example of defining coordinate constraints using pa.Coordinate and validating an xarray Dataset. ```python "lat": pa.Coordinate( dtype=np.float64, checks=pa.Check.in_range(-90, 90), ), "lon": pa.Coordinate( dtype=np.float64, checks=pa.Check.in_range(-180, 180), ), }, ) ds = xr.Dataset({...}) validated = schema.validate(ds) ``` -------------------------------- ### Define Schemas with Format Configuration Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/data_format_conversion.md Demonstrates how to subclass DataFrameModel to specify input and output formats using the Config class. This allows the @check_types decorator to handle serialization automatically. ```python import pandera.pandas as pa from pandera.typing import DataFrame, Series class InSchema(pa.DataFrameModel): str_col: Series[str] = pa.Field(unique=True, isin=[*"abcd"]) int_col: Series[int] class OutSchema(InSchema): float_col: pa.typing.Series[float] class InSchemaParquet(InSchema): class Config: from_format = "parquet" class OutSchemaDict(OutSchema): class Config: to_format = "dict" to_format_kwargs = {"orient": "records"} ``` -------------------------------- ### Defining the `coerce_value` Method Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/dtypes.md Details the importance of the `coerce_value` method for individual value coercion and provides an example from the `pandera.engines.pandas_engine.Category` class. ```APIDOC ## Defining the `coerce_value` Method ### Description This section explains the necessity of implementing the `coerce_value` method within custom Pandera data types. This method is crucial for Pandera to correctly handle and report errors during the coercion of individual values to the specified data type. It provides an example from the `pandera.engines.pandas_engine.Category` class. ### Method Implementation of the `coerce_value` method within a custom `DataType` class. ### Endpoint N/A (This is a library usage example) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from typing import Any # Assume Category class inherits from a base Pandera DataType # class Category(pandera.dtypes.DataType): # def __init__(self, categories, ordered=False): # self.categories = categories # self.ordered = ordered # self.type = 'category' # Example attribute # def coerce_value(self, value: Any) -> Any: # """Coerce an value to a particular type.""" # if value not in self.categories: # type: ignore # raise TypeError( # f"value {value} cannot be coerced to type {self.type}" # ) # return value ``` ### Response #### Success Response (200) Provides the source code for the `coerce_value` method in `pandera.engines.pandas_engine.Category`, demonstrating how to check if a value exists within defined categories before returning it. #### Response Example ```python def coerce_value(self, value: Any) -> Any: """Coerce an value to a particular type.""" if value not in self.categories: # type: ignore raise TypeError( f"value {value} cannot be coerced to type {self.type}" ) return value ``` ``` -------------------------------- ### Runtime Backend Re-registration Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/narwhals_backend.md Shows how updating the configuration after schema validation triggers an automatic re-registration of the backend. ```python import pandera.polars as pa schema = pa.DataFrameSchema({"age": pa.Column(int)}) schema.validate(df) # uses native Polars backend (default) pa.config.set_config(use_narwhals_backend=True) # UserWarning: Re-registered pandera backends after use_narwhals_backend changed. schema.validate(df) # same schema object, now validated by the Narwhals backend ``` -------------------------------- ### Using Parsers for Data Preprocessing Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/xarray_guide/checks_and_parsers.md Demonstrates how to use Parser objects to fill missing values and rename DataArrays before schema validation. This ensures data is in the desired format prior to checks. ```python schema = pa.DataArraySchema( parsers=[ pa.Parser(lambda da: da.fillna(0)), pa.Parser(lambda da: da.rename("cleaned")), ], checks=pa.Check(lambda da: float(da.min()) >= 0), ) da_messy = xr.DataArray([1.0, np.nan, 3.0], dims="x", name="raw") validated = schema.validate(da_messy) print(f"name: {validated.name}, values: {validated.values}") ``` -------------------------------- ### Schema with unsatisfiable checks Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/data_synthesis_strategies.md Example demonstrating an Unsatisfiable exception raised due to nonsensical constraints. ```python import hypothesis schema_multiple_checks = pa.DataFrameSchema({ "column1": pa.Column( float, checks=[ # nonsensical constraints pa.Check.gt(0), pa.Check.lt(-10), ] ) }) try: schema_multiple_checks.example(size=3) except Exception as e: print(e) ``` -------------------------------- ### Define Schemas using Class Inheritance Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/notebooks/try_pandera.ipynb Demonstrates how to extend a base Pandera DataFrameModel to create a new schema with additional fields, facilitating validation across different stages of data transformation. ```python class Schema(pa.DataFrameModel): item: Series[str] = pa.Field(isin=["apple", "orange"], coerce=True) price: Series[float] = pa.Field(gt=0, coerce=True) class TransformedSchema(Schema): expiry: Series[pd.Timestamp] = pa.Field(coerce=True) ``` -------------------------------- ### Define Coordinate Schema for Datetime Source: https://github.com/unionai-oss/pandera/blob/main/specs/xarray.md Example of creating a Coordinate schema for datetime values, specifying the expected data type as 'datetime64[ns]'. ```python time_coord = pa.Coordinate(dtype="datetime64[ns]") ``` -------------------------------- ### Built-in Check Registration Example Source: https://github.com/unionai-oss/pandera/blob/main/specs/optimized-strategies.md Illustrates how built-in checks are registered with their corresponding strategy functions. ```python from pandera.api.pandas.registry import register_builtin_check from pandera.strategies import pandas_strategies as st @register_builtin_check(strategy=st.gt_strategy, ...) def greater_than(...): ... ``` -------------------------------- ### Import Pandera GeoPandas module Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/geopandas.md Import the entry point for GeoPandas validation. ```python import pandera.geopandas as pg ``` -------------------------------- ### Defining a custom check strategy Source: https://github.com/unionai-oss/pandera/blob/main/docs/source/data_synthesis_strategies.md Example of defining a custom check and its corresponding strategy for in-range values. ```python check = pa.Check(lambda x: x.between(0, 100)) ``` ```python import pandera.strategies.pandas_strategies as st def in_range_strategy(pandera_dtype, strategy=None): if strategy is None: # handle base strategy case return st.floats(min_value=min_val, max_value=max_val).map( # the map isn't strictly necessary, but shows an example of # using the pandera_dtype argument strategies.to_numpy_dtype(pandera_dtype).type ) # handle chained strategy case return strategy.filter(lambda val: 0 <= val <= 10) check = pa.Check(lambda x: x.between(0, 100), strategy=in_range_strategy) ```