### Install Dependencies Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Installs project dependencies using poetry. ```bash make install ``` ```bash poetry install --all-groups --all-extras ``` -------------------------------- ### Precision Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Examples of using the 'precision' parameter with approximate equality checks. ```python # Allow 0.01 difference between float columns assert_approx_df_equality(df1, df2, precision=0.01) # Disable approximate comparison (use exact) assert_approx_df_equality(df1, df2, precision=0) ``` -------------------------------- ### Formats Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example demonstrating how to customize error message appearance using the 'formats' parameter. ```python from chispa import FormattingConfig, Color, Style formats = FormattingConfig( mismatched_rows={"color": Color.LIGHT_RED, "style": Style.BOLD}, matched_rows={"color": Color.LIGHT_GREEN}, mismatched_cells={"color": Color.YELLOW, "style": [Style.BOLD, Style.UNDERLINE]}, matched_cells={"color": Color.BLUE} ) assert_df_equality(df1, df2, formats=formats) ``` -------------------------------- ### ignore_metadata example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example demonstrating how to use ignore_metadata=True to compare schemas with different metadata. ```python # These schemas are equal with ignore_metadata=True schema1 = StructType([ StructField("id", IntegerType(), True, metadata={"description": "User ID"}) ]) schema2 = StructType([ StructField("id", IntegerType(), True, metadata={"description": "Different text"}) ]) assert_df_equality(df1, df2, ignore_metadata=True) ``` -------------------------------- ### ignore_columns example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example demonstrating how to use ignore_columns to exclude specific columns from comparison. ```python # Ignore id (auto-generated) and created_at (timestamp) assert_df_equality( actual, expected, ignore_columns=["id", "created_at"] ) ``` -------------------------------- ### ignore_nullable example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example demonstrating how to use ignore_nullable=True to compare schemas that differ only in nullability. ```python # These schemas are equal with ignore_nullable=True schema1 = StructType([ StructField("id", IntegerType(), True), # nullable = True StructField("name", StringType(), True) ]) schema2 = StructType([ StructField("id", IntegerType(), False), # nullable = False StructField("name", StringType(), False) ]) assert_df_equality(df1, df2, ignore_nullable=True) ``` -------------------------------- ### allow_nan_equality example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example demonstrating how to use allow_nan_equality=True to treat NaN values as equal. ```python import math # df1 has row: (1.0, math.nan) ``` -------------------------------- ### Custom Error Format Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Placeholder for an example demonstrating custom error formatting. -------------------------------- ### Flexible Schema, Strict Data Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example allowing schema flexibility while ensuring data matches exactly. ```python # Allow schema flexibility, but data must match exactly assert_df_equality( df1, df2, ignore_nullable=True, ignore_metadata=True, ignore_column_order=True ) ``` -------------------------------- ### Transform and Compare Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example combining transformations (filtering, sorting) with DataFrame comparison. ```python # Filter, sort, then compare assert_df_equality( df1, df2, transforms=[ lambda df: df.filter(df.status == "active"), lambda df: df.orderBy("id", "name") ] ) ``` -------------------------------- ### Underline Cells Example (Superseded) Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example showing the 'underline_cells' parameter, noting it's largely superseded by the 'formats' parameter. ```python # For more control, use formats parameter instead formats = FormattingConfig( mismatched_cells={"style": Style.UNDERLINE} ) assert_df_equality(df1, df2, formats=formats) ``` -------------------------------- ### ignore_column_order example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example demonstrating how to use ignore_column_order=True to compare DataFrames with different column orders. ```python # df1 has columns: name, age, email # df2 has columns: email, name, age # With ignore_column_order=True, both become: age, email, name assert_df_equality(df1, df2, ignore_column_order=True) ``` -------------------------------- ### Complete Example - Option 1: Direct formatting Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md A complete example demonstrating how to define and use custom formatting directly with the assert_df_equality function. ```python from chispa import ( assert_df_equality, FormattingConfig, Color, Style, Chispa ) # Define custom formatting for your test suite my_formats = FormattingConfig( mismatched_rows=( "color": Color.LIGHT_RED, "style": Style.BOLD ), matched_rows=( "color": Color.LIGHT_GREEN, "style": Style.BOLD ), mismatched_cells=( "color": Color.LIGHT_YELLOW, "style": [Style.BOLD, Style.UNDERLINE] ), matched_cells=( "color": Color.LIGHT_BLUE ) ) # Option 1: Use directly with functions def test_with_direct_formatting(): assert_df_equality(df1, df2, formats=my_formats) ``` -------------------------------- ### Spark Session Import Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Example of importing the shared Spark session for testing. ```python from .spark import spark ``` -------------------------------- ### Transforms Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example of using the 'transforms' parameter to apply custom logic before DataFrame comparison. ```python # Filter rows, sort, and cast columns assert_df_equality( actual, expected, transforms=[ lambda df: df.filter(df.age > 18), # Filter lambda df: df.orderBy("id"), # Sort lambda df: df.select(F.col("name").cast(StringType())) # Cast ] ) ``` -------------------------------- ### ignore_row_order example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example demonstrating how to use ignore_row_order=True to compare DataFrames with different row orders. ```python # These DataFrames are equal with ignore_row_order=True # df1 rows: (alice, 25), (bob, 30) # df2 rows: (bob, 30), (alice, 25) assert_df_equality(df1, df2, ignore_row_order=True) ``` -------------------------------- ### Build & Serve Documentation Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Builds and serves the project documentation using mkdocs. ```bash make docs ``` -------------------------------- ### Selective Column Comparison Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example for comparing only specific columns and ignoring others. ```python # Compare only specific columns, ignore generated columns assert_df_equality( df1, df2, ignore_columns=["id", "created_at", "updated_at"] ) ``` -------------------------------- ### Custom Colors for Accessibility/Preference Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example of using `FormattingConfig` and `Color` to customize the appearance of mismatched and matched rows in DataFrame comparisons. ```python from chispa import FormattingConfig, Color formats = FormattingConfig( mismatched_rows={"color": Color.LIGHT_YELLOW}, matched_rows={"color": Color.LIGHT_CYAN} ) assert_df_equality(df1, df2, formats=formats) ``` -------------------------------- ### Ignore Type Metadata Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example for ignoring nullable and metadata differences when types match. ```python # When metadata differs but types match assert_df_equality( df1, df2, ignore_nullable=True, ignore_metadata=True ) ``` -------------------------------- ### Style Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-formatting.md Example of using the Style enum with FormattingConfig. ```python from chispa import Style, FormattingConfig # Using enum member directly formats = FormattingConfig( mismatched_cells={"color": "red", "style": Style.UNDERLINE} ) # Multiple styles as list formats = FormattingConfig( matched_rows={"color": "blue", "style": [Style.BOLD, Style.UNDERLINE]} ) ``` -------------------------------- ### Approximate with NaN Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example for approximate numeric comparison that also handles NaN values. ```python # Approximate numeric comparison with NaN handling assert_approx_df_equality( df1, df2, precision=0.01, allow_nan_equality=True ) ``` -------------------------------- ### Multiple Styles - Single style string Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of specifying a single style string. ```python {"style": "bold"} ``` -------------------------------- ### FormattingConfig Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-formatting.md Example demonstrating how to create and use FormattingConfig objects with both Format objects and dictionaries, and how to apply them to DataFrame equality assertions. ```python from chispa import FormattingConfig, Color, Style # Using Format objects (recommended) formats = FormattingConfig( mismatched_rows={"color": Color.LIGHT_YELLOW}, matched_rows={"color": Color.CYAN, "style": Style.BOLD}, mismatched_cells={"color": Color.PURPLE}, matched_cells={"color": Color.BLUE} ) # Or using dictionaries with string names formats = FormattingConfig( mismatched_rows={"color": "light_yellow"}, matched_rows={"color": "cyan", "style": "bold"}, mismatched_cells={"color": "purple"}, matched_cells={"color": "blue"} ) # Use with assertions from chispa import assert_df_equality assert_df_equality(df1, df2, formats=formats) # Or with Chispa class from chispa import Chispa chispa = Chispa(formats=formats) chispa.assert_df_equality(df1, df2) ``` -------------------------------- ### Using Chispa Instance Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of initializing Chispa with custom formats and using it for DataFrame equality assertions. ```python chispa_instance = Chispa(formats=my_formats) def test_with_chispa_instance(): chispa_instance.assert_df_equality(df1, df2) chispa_instance.assert_df_equality(df3, df4) ``` -------------------------------- ### Run Pre-commit Hooks Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Executes all pre-commit hooks. ```bash poetry run pre-commit run -a ``` -------------------------------- ### Install chispa Source: https://github.com/mrpowers/chispa/blob/main/README.md Install the latest version of chispa using pip. ```sh pip install chispa ``` -------------------------------- ### assert_schema_equality usage example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-schema-functions.md Example demonstrating how to use assert_schema_equality to verify matching schemas and schemas with ignored nullable differences. ```python from pyspark.sql.types import StructType, StructField, StringType, IntegerType from chispa import assert_schema_equality # Create two matching schemas s1 = StructType([ StructField("name", StringType(), True), StructField("age", IntegerType(), True) ]) s2 = StructType([ StructField("name", StringType(), True), StructField("age", IntegerType(), True) ]) # Verify schemas match assert_schema_equality(s1, s2) # Ignore nullable differences s3 = StructType([ StructField("name", StringType(), False), # Different nullable StructField("age", IntegerType(), True) ]) assert_schema_equality(s1, s3, ignore_nullable=True) ``` -------------------------------- ### assert_basic_rows_equality Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-rows-functions.md Example demonstrating how to use assert_basic_rows_equality with and without custom formatting. ```python from pyspark.sql import SparkSession, Row from chispa import assert_basic_rows_equality, FormattingConfig, Color spark = SparkSession.builder.appName("example").master("local").getOrCreate() # Create DataFrames and collect rows df1 = spark.createDataFrame( [("alice", 25), ("bob", 30)], ["name", "age"] ) df2 = spark.createDataFrame( [("alice", 25), ("bob", 30)], ["name", "age"] ) rows1 = df1.collect() rows2 = df2.collect() # Simple equality check assert_basic_rows_equality(rows1, rows2) # With custom formatting formats = FormattingConfig( matched_cells={"color": Color.CYAN}, mismatched_cells={"color": Color.YELLOW} ) assert_basic_rows_equality(rows1, rows2, formats=formats) ``` -------------------------------- ### isnan usage example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-helper-functions.md Example demonstrating the usage of the isnan function with various inputs. ```python from chispa.number_helpers import isnan import math assert isnan(float('nan')) == True assert isnan(1.5) == False assert isnan("string") == False # No exception raised assert isnan(None) == False ``` -------------------------------- ### Multiple Styles - List of style strings Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of specifying multiple styles using a list of style strings. ```python {"style": ["bold", "underline"]} ``` -------------------------------- ### Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-chispa-class.md An example demonstrating how to create a Chispa instance with custom formatting and use it for DataFrame assertions. ```python from pyspark.sql import SparkSession from chispa import Chispa, FormattingConfig, Color, Style spark = SparkSession.builder.appName("example").master("local").getOrCreate() # Create custom formatting configuration custom_formats = FormattingConfig( mismatched_rows={"color": Color.LIGHT_YELLOW}, matched_rows={"color": Color.CYAN, "style": Style.BOLD}, mismatched_cells={"color": Color.PURPLE}, matched_cells={"color": Color.BLUE} ) # Create Chispa instance with custom formatting chispa = Chispa(formats=custom_formats) # Use the instance for assertions df1 = spark.createDataFrame( [("alice", 25), ("bob", 30)], ["name", "age"] ) df2 = spark.createDataFrame( [("alice", 25), ("bob", 30)], ["name", "age"] ) # All assertions from this instance use custom formatting chispa.assert_df_equality(df1, df2) chispa.assert_df_equality(df1, df2, ignore_column_order=True) ``` -------------------------------- ### Color Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-formatting.md Example of using the Color enum with FormattingConfig. ```python from chispa import Color, FormattingConfig # Using enum member directly formats = FormattingConfig( matched_cells={"color": Color.BLUE} ) # Or as string (case-insensitive) formats = FormattingConfig( matched_cells={"color": "blue"} ) ``` -------------------------------- ### Format Specification - Dictionary Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of specifying formatting using dictionaries with 'color' and 'style' keys. ```python formats = FormattingConfig( mismatched_rows={"color": "light_yellow"}, matched_rows={"color": "cyan", "style": "bold"} ) ``` -------------------------------- ### Run All Tests Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Runs all project tests using pytest with coverage. ```bash make test ``` ```bash poetry run pytest -vvv ``` -------------------------------- ### Transformation Pipeline Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md Example demonstrating the use of the `transforms` parameter to filter and sort DataFrames before comparison. ```python assert_df_equality( df1, df2, transforms=[ lambda df: df.filter(df.age > 18), lambda df: df.orderBy("name") ] ) ``` -------------------------------- ### assert_column_equality Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-column-functions.md Example demonstrating how to use assert_column_equality to verify that two columns in a DataFrame are identical. ```python from pyspark.sql import SparkSession from chispa import assert_column_equality spark = SparkSession.builder.appName("example").master("local").getOrCreate() # Create a DataFrame with two columns data = [ ("jose", "jose"), ("li", "li"), ("luisa", "luisa"), (None, None) ] df = spark.createDataFrame(data, ["original", "cleaned"]) # Verify columns are equal assert_column_equality(df, "original", "cleaned") ``` -------------------------------- ### Ignore Structural Differences Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Example for ignoring row and column order differences during DataFrame comparison. ```python # When row/column order differs assert_df_equality( df1, df2, ignore_row_order=True, ignore_column_order=True ) ``` -------------------------------- ### Format Specification - Format object Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of specifying formatting using Format objects, recommended for IDE type hints. ```python from chispa import FormattingConfig, Format, Color, Style formats = FormattingConfig( mismatched_rows=Format(color=Color.LIGHT_YELLOW), matched_rows=Format(color=Color.CYAN, style=[Style.BOLD]) ) ``` -------------------------------- ### ColumnsNotEqualError Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/errors.md Example demonstrating how ColumnsNotEqualError is raised and caught. ```python from pyspark.sql import SparkSession from chispa import assert_column_equality, ColumnsNotEqualError spark = SparkSession.builder.appName("example").master("local").getOrCreate() data = [("alice", "alice"), ("bob", "charlie")] df = spark.createDataFrame(data, ["actual", "expected"]) try: assert_column_equality(df, "actual", "expected") except ColumnsNotEqualError as e: print(f"Columns not equal: {e}") ``` -------------------------------- ### are_rows_approx_equal usage example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-helper-functions.md Example demonstrating the usage of are_rows_approx_equal with different precision values. ```python from pyspark.sql import Row from chispa.row_comparer import are_rows_approx_equal r1 = Row(value=1.1, name="test") r2 = Row(value=1.15, name="test") # Within precision 0.1 assert are_rows_approx_equal(r1, r2, precision=0.1) == True # Beyond precision 0.1 assert are_rows_approx_equal(r1, r2, precision=0.01) == False ``` -------------------------------- ### DataFramesNotEqualError Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/errors.md Example demonstrating how DataFramesNotEqualError is raised and caught. ```python from pyspark.sql import SparkSession from chispa import assert_df_equality, DataFramesNotEqualError spark = SparkSession.builder.appName("example").master("local").getOrCreate() df1 = spark.createDataFrame([("alice", 25), ("bob", 30)], ["name", "age"]) df2 = spark.createDataFrame([("alice", 25), ("bob", 40)], ["name", "age"]) try: assert_df_equality(df1, df2) except DataFramesNotEqualError as e: print(f"DataFrames not equal:\n{e}") ``` -------------------------------- ### Install chispa with Poetry Source: https://github.com/mrpowers/chispa/blob/main/README.md Add chispa as a development dependency using Poetry. ```sh poetry add chispa --group dev ``` -------------------------------- ### Ruff Lint & Format Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Commands to check and format code using Ruff. ```bash poetry run ruff check . ``` ```bash poetry run ruff format . ``` -------------------------------- ### Build Documentation Test Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Checks if documentation builds without warnings. ```bash make docs-test ``` -------------------------------- ### assert_generic_rows_equality Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-rows-functions.md Example demonstrating the usage of assert_generic_rows_equality with approximate row equality. ```python from pyspark.sql import SparkSession, Row from chispa import assert_generic_rows_equality from chispa.row_comparer import are_rows_approx_equal spark = SparkSession.builder.appName("example").master("local").getOrCreate() # Create Row objects rows1 = [ Row(value=1.1, letter="a"), Row(value=2.2, letter="b") ] rows2 = [ Row(value=1.05, letter="a"), Row(value=2.13, letter="b") ] ``` -------------------------------- ### Multiple Styles - List of Style enums Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of specifying multiple styles using a list of Style enums. ```python {"style": [Style.BOLD, Style.UNDERLINE]} ``` -------------------------------- ### are_rows_equal_enhanced usage example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-helper-functions.md Example demonstrating the usage of are_rows_equal_enhanced with and without NaN equality. ```python from pyspark.sql import Row from chispa.row_comparer import are_rows_equal_enhanced import math r1 = Row(value=math.nan, name="test") r2 = Row(value=math.nan, name="test") # Without NaN equality assert are_rows_equal_enhanced(r1, r2, allow_nan_equality=False) == False # With NaN equality assert are_rows_equal_enhanced(r1, r2, allow_nan_equality=True) == True ``` -------------------------------- ### assert_approx_column_equality Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-column-functions.md Example demonstrating how to use assert_approx_column_equality to verify approximate equality between two numeric columns. ```python from pyspark.sql import SparkSession from chispa import assert_approx_column_equality spark = SparkSession.builder.appName("example").master("local").getOrCreate() # Create a DataFrame with floating point columns data = [ (1.1, 1.1), (2.2, 2.15), (3.3, 3.37), (None, None) ] df = spark.createDataFrame(data, ["computed", "expected"]) # Verify columns are approximately equal (within 0.1) assert_approx_column_equality(df, "computed", "expected", 0.1) ``` -------------------------------- ### Usage example for are_datatypes_equal Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-schema-functions.md Example demonstrating the usage of are_datatypes_equal with simple primitive types and complex nested types. ```python from pyspark.sql.types import ( ArrayType, StringType, StructType, StructField, IntegerType ) from chispa import are_datatypes_equal # Simple primitive types dt1 = StringType() dt2 = StringType() assert are_datatypes_equal(dt1, dt2) # Complex nested types array_struct = ArrayType( StructType([ StructField("id", IntegerType(), True), StructField("name", StringType(), True) ]) ) assert are_datatypes_equal(array_struct, array_struct) ``` -------------------------------- ### are_schemas_equal usage example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-schema-functions.md Example showing how to use are_schemas_equal to check if two schemas are equal. ```python from pyspark.sql.types import StructType, StructField, StringType, IntegerType from chispa import are_schemas_equal s1 = StructType([ StructField("name", StringType(), True), StructField("age", IntegerType(), True) ]) s2 = StructType([ StructField("name", StringType(), True), StructField("age", IntegerType(), True) ]) if are_schemas_equal(s1, s2): print("Schemas match") ``` -------------------------------- ### Backward Compatibility with DefaultFormats Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Demonstrates the use of the deprecated DefaultFormats class and its recommended replacement, FormattingConfig, with examples of color and style configurations. ```python from chispa import DefaultFormats # Deprecated # This still works but shows DeprecationWarning formats = DefaultFormats( mismatched_rows=["light_yellow"], matched_rows=["cyan", "bold"] ) # Use FormattingConfig instead from chispa import FormattingConfig, Color, Style formats = FormattingConfig( mismatched_rows={"color": "light_yellow"}, matched_rows={"color": "cyan", "style": "bold"} ) ``` -------------------------------- ### Run a Specific Test Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Executes a particular test case. ```bash poetry run pytest tests/test_dataframe_comparer.py::describe_assert_df_equality::it_throws_with_schema_mismatches ``` -------------------------------- ### Usage with Assertions - Chispa class instance Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of initializing Chispa with custom formatting to be used for all assertions. ```python from chispa import Chispa, FormattingConfig, Color custom_formats = FormattingConfig( mismatched_rows={"color": "light_yellow"}, matched_rows={"color": "cyan"} ) chispa = Chispa(formats=custom_formats) # All assertions use custom formatting chispa.assert_df_equality(df1, df2) chispa.assert_df_equality(df3, df4) ``` -------------------------------- ### Coverage Report (HTML) Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Generates an HTML coverage report. ```bash make test-cov-html ``` -------------------------------- ### are_structfields_equal usage example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-schema-functions.md Example demonstrating the usage of are_structfields_equal to compare StructFields, including ignoring nullability. ```python from pyspark.sql.types import StructField, StringType, IntegerType from chispa import are_structfields_equal sf1 = StructField("name", StringType(), True) sf2 = StructField("name", StringType(), True) sf3 = StructField("name", StringType(), False) assert are_structfields_equal(sf1, sf2) # True assert are_structfields_equal(sf1, sf3) == False # False assert are_structfields_equal(sf1, sf3, ignore_nullability=True) # True ``` -------------------------------- ### assert_approx_df_equality function signature Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md Signature for the assert_approx_df_equality function, showing its parameters. ```python def assert_approx_df_equality( df1: DataFrame, df2: DataFrame, precision: float, # REQUIRED ignore_nullable: bool = False, transforms: list[Callable] | None = None, allow_nan_equality: bool = False, ignore_column_order: bool = False, ignore_row_order: bool = False, ignore_columns: list[str] | None = None, formats: FormattingConfig | None = None, ) -> None ``` -------------------------------- ### Usage with Assertions - Module-level functions Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of using custom formatting with module-level assert_df_equality function. ```python from chispa import assert_df_equality, FormattingConfig, Color custom_formats = FormattingConfig( mismatched_rows={"color": "light_yellow"}, matched_rows={"color": "cyan"} ) assert_df_equality(df1, df2, formats=custom_formats) ``` -------------------------------- ### Usage Example for nan_safe_equality Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-helper-functions.md Demonstrates how to use nan_safe_equality to compare numbers, including NaN values. ```python from chispa.number_helpers import nan_safe_equality import math assert nan_safe_equality(1.5, 1.5) == True assert nan_safe_equality(math.nan, math.nan) == True assert nan_safe_equality(1.5, 2.0) == False ``` -------------------------------- ### Mypy (Type Check) Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Performs type checking using mypy. ```bash poetry run mypy chispa ``` -------------------------------- ### Coverage Report (XML) Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Generates an XML coverage report. ```bash make test-cov-xml ``` -------------------------------- ### Example of ignoring nullability property Source: https://github.com/mrpowers/chispa/blob/main/README.md A detailed example demonstrating how to ignore the nullable property when comparing DataFrames, including schema and DataFrame creation. ```python def ignore_nullable_property(): s1 = StructType([ StructField("name", StringType(), True), StructField("age", IntegerType(), True)]) df1 = spark.createDataFrame([("juan", 7), ("bruna", 8)], s1) s2 = StructType([ StructField("name", StringType(), True), StructField("age", IntegerType(), False)]) df2 = spark.createDataFrame([("juan", 7), ("bruna", 8)], s2) assert_df_equality(df1, df2) ``` -------------------------------- ### Schema Equality Assertion Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/errors.md Demonstrates how to use assert_schema_equality and catch SchemasNotEqualError when schema types differ. ```python from pyspark.sql.types import StructType, StructField, StringType, IntegerType from chispa import assert_schema_equality, SchemasNotEqualError s1 = StructType([ StructField("name", StringType(), True), StructField("age", IntegerType(), True) ]) s2 = StructType([ StructField("name", StringType(), True), StructField("age", StringType(), True) # Different type ]) try: assert_schema_equality(s1, s2) except SchemasNotEqualError as e: print(f"Schemas not equal:\n{e}") ``` -------------------------------- ### Run a Single Test File Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Executes tests within a specific file. ```bash poetry run pytest tests/test_dataframe_comparer.py ``` -------------------------------- ### Usage with Assertions - Pytest fixture Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md Example of providing custom formatting via a pytest fixture. ```python import pytest from chispa import FormattingConfig, Color, Style @pytest.fixture() def chispa_formats(): return FormattingConfig( mismatched_rows={"color": Color.LIGHT_YELLOW}, matched_rows={"color": Color.CYAN, "style": Style.BOLD}, mismatched_cells={"color": Color.PURPLE}, matched_cells={"color": Color.BLUE} ) def test_with_custom_formats(chispa_formats): from chispa import assert_df_equality assert_df_equality(df1, df2, formats=chispa_formats) ``` -------------------------------- ### Usage Example for nan_safe_approx_equality Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-helper-functions.md Illustrates the usage of nan_safe_approx_equality for approximate number comparison, handling NaN and Decimal precision. ```python from chispa.number_helpers import nan_safe_approx_equality import math from decimal import Decimal # Within tolerance assert nan_safe_approx_equality(1.1, 1.15, 0.1) == True # Beyond tolerance assert nan_safe_approx_equality(1.1, 1.5, 0.1) == False # NaN handling assert nan_safe_approx_equality(math.nan, math.nan, 0.1) == True # With Decimal precision assert nan_safe_approx_equality(1.1, 1.15, Decimal("0.1")) == True ``` -------------------------------- ### Usage Example Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-dataframe-functions.md Demonstrates how to use assert_approx_df_equality with and without NaN handling. ```python from pyspark.sql import SparkSession from chispa import assert_approx_df_equality spark = SparkSession.builder.appName("example").master("local").getOrCreate() # Create DataFrames with floating point data df1 = spark.createDataFrame( [(1.1, "a"), (2.2, "b"), (3.3, "c")], ["value", "letter"] ) df2 = spark.createDataFrame( [(1.05, "a"), (2.13, "b"), (3.3, "c")], ["value", "letter"] ) # Verify approximate equality within 0.1 assert_approx_df_equality(df1, df2, 0.1) # With NaN handling import math df3 = spark.createDataFrame( [(1.1, "a"), (math.nan, "b")], ["value", "letter"] ) df4 = spark.createDataFrame( [(1.1, "a"), (math.nan, "b")], ["value", "letter"] ) assert_approx_df_equality(df3, df4, 0.1, allow_nan_equality=True) ``` -------------------------------- ### Full Check (Linting & Type Checking) Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Performs a full check including pre-commit hooks and mypy. ```bash make check ``` ```bash pre-commit run --all-files ``` -------------------------------- ### Format.from_list Class Method Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-formatting.md The signature and example for the Format.from_list class method, which creates a Format instance from a list of color and style names. ```python @classmethod def from_list(cls, values: list[str]) -> Format # Example format_obj = Format.from_list(["blue", "bold", "underline"]) ``` -------------------------------- ### Chispa Class Initialization Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md The `__init__` method of the `Chispa` class, which accepts a `FormattingConfig` object to store formats for instance methods. ```python class Chispa: def __init__(self, formats: FormattingConfig | None = None) -> None: # Stores formats for use in all instance methods ... ``` -------------------------------- ### Column Comparison Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md Example of comparing two columns within the same DataFrame using `assert_column_equality`. ```python from chispa import assert_column_equality # Verify two columns in same DataFrame are equal df = df.withColumn("clean_name", clean(F.col("name"))) assert_column_equality(df, "clean_name", "expected_clean_name") ``` -------------------------------- ### Format.from_dict Class Method Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-formatting.md The signature and example for the Format.from_dict class method, which creates a Format instance from a dictionary. ```python @classmethod def from_dict(cls, format_dict: dict[str, str | list[str]]) -> Format # Example format_obj = Format.from_dict({"color": "blue", "style": "bold"}) ``` -------------------------------- ### Basic DataFrame Comparison Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md A fundamental example of comparing two DataFrames for equality using `assert_df_equality`. ```python from chispa import assert_df_equality actual = process_data(input_df) expected = spark.createDataFrame(...) assert_df_equality(actual, expected) ``` -------------------------------- ### Usage Example for assert_df_equality Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/api-reference-dataframe-functions.md Demonstrates various ways to use assert_df_equality, including simple equality checks, ignoring column/row order, ignoring specific columns, and applying transformations. ```python from pyspark.sql import SparkSession from chispa import assert_df_equality spark = SparkSession.builder.appName("example").master("local").getOrCreate() # Create two DataFrames df1 = spark.createDataFrame( [("alice", 25), ("bob", 30)], ["name", "age"] ) df2 = spark.createDataFrame( [("alice", 25), ("bob", 30)], ["name", "age"] ) # Simple equality check assert_df_equality(df1, df2) # Ignore column order df3 = spark.createDataFrame( [(25, "alice"), (30, "bob")], ["age", "name"] ) assert_df_equality(df1, df3, ignore_column_order=True) # Ignore row order df4 = spark.createDataFrame( [("bob", 30), ("alice", 25)], ["name", "age"] ) assert_df_equality(df1, df4, ignore_row_order=True) # Ignore specific columns assert_df_equality(df1, df2, ignore_columns=["age"]) # Apply transformations before comparison assert_df_equality( df1, df2, transforms=[lambda df: df.filter(df.age > 20)] ) ``` -------------------------------- ### Transformation Pipeline Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/INDEX.md Example of applying transformations to DataFrames before comparison using the 'transforms' parameter. ```python assert_df_equality( df1, df2, transforms=[ lambda df: df.filter(...), lambda df: df.select(...), ] ) ``` -------------------------------- ### Custom formatting via conftest.py fixture Source: https://github.com/mrpowers/chispa/blob/main/README.md Example of defining custom formats in conftest.py and injecting them via a pytest fixture. ```python @pytest.fixture() def chispa_formats(): return FormattingConfig( mismatched_rows={"color": "light_yellow"}, matched_rows={"color": "cyan", "style": "bold"}, mismatched_cells={"color": "purple"}, matched_cells={"color": "blue"}, ) def test_shows_assert_basic_rows_equality(chispa_formats): ... assert_basic_rows_equality(df1.collect(), df2.collect(), formats=chispa_formats) ``` -------------------------------- ### Chispa Class assert_df_equality Method Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/parameter-guide.md The `assert_df_equality` method of the `Chispa` class, which uses the instance's saved formatting configuration. ```python def assert_df_equality( self, df1: DataFrame, df2: DataFrame, ignore_nullable: bool = False, transforms: list[Callable] | None = None, allow_nan_equality: bool = False, ignore_column_order: bool = False, ignore_row_order: bool = False, underline_cells: bool = False, ignore_metadata: bool = False, ignore_columns: list[str] | None = None, ) -> None: # Same parameters as module function, but formats comes from self.formats ... ``` -------------------------------- ### Main Entry Point Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md Import the chispa library for general use. ```python import chispa ``` -------------------------------- ### Allow NaN equality Source: https://github.com/mrpowers/chispa/blob/main/README.md Examples of using assert_df_equality with and without the allow_nan_equality=True flag. ```python # default behavior remains strict (NaN != NaN) assert_df_equality(df1, df2) # opt in to treat NaN values as equal assert_df_equality(df1, df2, allow_nan_equality=True) ``` -------------------------------- ### Default Formatting Explanation Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/configuration.md The default configuration for FormattingConfig, optimized for terminal clarity. ```python FormattingConfig( mismatched_rows=Format(Color.RED), matched_rows=Format(Color.BLUE), mismatched_cells=Format(Color.RED, [Style.UNDERLINE]), matched_cells=Format(Color.BLUE), ) ``` -------------------------------- ### Catching Specific Errors Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/errors.md Example of catching specific Chispa errors like SchemasNotEqualError and DataFramesNotEqualError. ```python from chispa import ( assert_df_equality, ColumnsNotEqualError, DataFramesNotEqualError, SchemasNotEqualError ) try: assert_df_equality(df1, df2) except SchemasNotEqualError as e: print(f"Schema mismatch: {e}") except DataFramesNotEqualError as e: print(f"Data mismatch: {e}") ``` -------------------------------- ### Format Class Creation Methods Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/types.md The Format class can be created in multiple ways. ```python # Direct instantiation from chispa import Format, Color, Style # With enum values fmt1 = Format(color=Color.BLUE, style=[Style.BOLD]) # From dictionary fmt2 = Format.from_dict({"color": "blue", "style": "bold"}) # From list fmt3 = Format.from_list(["blue", "bold"]) ``` -------------------------------- ### Run Tests Without Coverage Source: https://github.com/mrpowers/chispa/blob/main/AGENTS.md Executes tests without generating a coverage report. ```bash poetry run pytest tests ``` -------------------------------- ### Custom formatting with FormattingConfig Source: https://github.com/mrpowers/chispa/blob/main/README.md Example of customizing printed error messages using FormattingConfig with string color names. ```python from chispa import FormattingConfig formats = FormattingConfig( mismatched_rows={"color": "light_yellow"}, matched_rows={"color": "cyan", "style": "bold"}, mismatched_cells={"color": "purple"}, matched_cells={"color": "blue"}, ) assert_basic_rows_equality(df1.collect(), df2.collect(), formats=formats) ``` -------------------------------- ### Additional public DataFrame comparison options Source: https://github.com/mrpowers/chispa/blob/main/README.md Example demonstrating the use of ignore_metadata, transforms, and underline_cells options in assert_df_equality. ```python assert_df_equality( actual_df, expected_df, transforms=[lambda df: df.orderBy("id")], ignore_metadata=True, underline_cells=True, ) ``` -------------------------------- ### Approximate Numeric Comparison Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md Example of using `assert_approx_df_equality` for comparing numeric columns with a specified tolerance. ```python from chispa import assert_approx_df_equality # Compare with 0.01 tolerance for float columns assert_approx_df_equality(actual, expected, 0.01) ``` -------------------------------- ### OutputFormat Usage Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/types.md Used with `print_schema_diff` to control how schema differences are displayed. ```python from chispa.schema_comparer import print_schema_diff, OutputFormat # Display as table print_schema_diff(schema1, schema2, False, False, OutputFormat.TABLE) # Display as tree print_schema_diff(schema1, schema2, False, False, OutputFormat.TREE) ``` -------------------------------- ### Custom Formatting Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md Shows how to customize the output formatting for DataFrame comparisons using `FormattingConfig` and the `Chispa` instance. ```python from chispa import Chispa, FormattingConfig, Color, Style formats = FormattingConfig( mismatched_rows={"color": Color.LIGHT_YELLOW}, matched_rows={"color": Color.CYAN, "style": Style.BOLD} ) chispa = Chispa(formats=formats) chispa.assert_df_equality(actual, expected) ``` -------------------------------- ### Recommended Imports for Common Use Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md A list of recommended imports for basic assertions, exception handling, and formatting in Chispa. ```python # Basic assertions from chispa import ( assert_df_equality, assert_approx_df_equality, assert_column_equality, assert_approx_column_equality ) # Exception handling from chispa import ( ColumnsNotEqualError, DataFramesNotEqualError, SchemasNotEqualError ) # Formatting from chispa import FormattingConfig, Color, Style # Advanced: Instance-based API from chispa import Chispa ``` -------------------------------- ### Formatting Configuration Source: https://github.com/mrpowers/chispa/blob/main/_autodocs/API-OVERVIEW.md Import classes for customizing error message formatting. ```python from chispa import FormattingConfig, Color, Style, Format ```