### Install Build Tools

Source: https://capitalone.github.io/datacompy/developer_instructions.html

Install or upgrade the necessary tools for building and uploading Python packages.

```bash
pip install --upgrade build wheel twine

```

--------------------------------

### Install Development Requirements

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Install the development requirements for the project, including dependencies for documentation generation.

```bash
pip install -e .[dev]
```

--------------------------------

### Install Build Tools

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Install or upgrade the necessary tools for building and uploading Python packages. Ensure you have the latest versions.

```bash
pip install --upgrade build wheel twine
```

--------------------------------

### Set up virtualenv for DataComPy

Source: https://capitalone.github.io/datacompy/_sources/install.rst.txt

Create a virtual environment, activate it, upgrade setuptools and pip, then install DataComPy. This method installs dependencies from PyPI.

```bash
virtualenv env
```

```bash
source env/bin/activate
```

```bash
pip install --upgrade setuptools pip
```

```bash
pip install datacompy
```

--------------------------------

### Install DataComPy

Source: https://capitalone.github.io/datacompy/index.html

Install DataComPy using pip or conda. For Spark or other backends, use the respective extras.

```bash
pip install datacompy
```

```bash
conda install datacompy
```

```bash
pip install datacompy[spark]
```

```bash
pip install datacompy[fugue]
```

```bash
pip install datacompy[snowflake]
```

--------------------------------

### Edgetest Output Example

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Example output from running edgetest, showing environment, test status, upgraded packages, and their versions.

```text
=============	================	===================	=================
Environment		Passing tests		Upgraded packages		Package version
=============		================		===================		=================
core			True			boto3				1.21.7
core			True			pandas				1.3.5
core			True			PyYAML				6.0
=============		================		===================		=================
```

--------------------------------

### Install DataComPy with Fugue Support

Source: https://capitalone.github.io/datacompy/_sources/fugue_usage.rst.txt

Install the DataComPy library with the necessary dependencies for Fugue integration using pip.

```bash
pip install datacompy[fugue]
```

--------------------------------

### Install DataComPy using pip

Source: https://capitalone.github.io/datacompy/_sources/install.rst.txt

Use this command for a basic installation of DataComPy via pip. A virtual environment is recommended.

```bash
pip install datacompy
```

--------------------------------

### Sample SparkSQLCompare Report Output

Source: https://capitalone.github.io/datacompy/spark_usage.html

Example of the string output generated by the report() method.

```text
DataComPy Comparison
--------------------

DataFrame Summary
-----------------

  DataFrame  Columns  Rows
0  Original        5     5
1       New        4     5

Column Summary
--------------

Number of columns in common: 4
Number of columns in Original but not in New: 1
Number of columns in New but not in Original: 0

Row Summary
-----------

Matched on: acct_id
Any duplicates on match values: No
Absolute Tolerance: 0
Relative Tolerance: 0
Number of rows in common: 4
Number of rows in Original but not in New: 1
Number of rows in New but not in Original: 1

Number of rows with some compared columns unequal: 4
Number of rows with all compared columns equal: 0

Column Comparison
-----------------

Number of columns compared with some values unequal: 3
Number of columns compared with all values equal: 1
Total number of values which compare unequal: 6

Columns with Unequal Values or Types
------------------------------------

       Column Original dtype New dtype  # Unequal  Max Diff  # Null Diff
0  dollar_amt        float64   float64          1    0.0500            0
2   float_fld        float64   float64          3    0.0005            2
1        name         object    object          2       NaN            0

Sample Rows with Unequal Values
-------------------------------

       acct_id  dollar_amt (Original)  dollar_amt (New)
0  10000001234                 123.45             123.4

       acct_id name (Original)            name (New)
0  10000001234  George Maharis  George Michael Bluth
3  10000001237      Bob Loblaw         Robert Loblaw

       acct_id  float_fld (Original)  float_fld (New)
0  10000001234            14530.1555        14530.155
1  10000001235                1.0000              NaN
2  10000001236                   NaN            1.000

Sample Rows Only in Original (First 10 Columns)
-----------------------------------------------

   acct_id_df1  dollar_amt_df1       name_df1  float_fld_df1 date_fld_df1  _merge_left
5  10000001239            1.05  Lucille Bluth            NaN   2017-01-01         True

Sample Rows Only in New (First 10 Columns)
------------------------------------------

   acct_id_df2  dollar_amt_df2          name_df2  float_fld_df2  _merge_right
4  10000001238            1.05  Loose Seal Bluth          111.0          True
```

--------------------------------

### Setup DataFrames for Comparison

Source: https://capitalone.github.io/datacompy/_sources/pandas_usage.rst.txt

Initialize two pandas DataFrames from string data using StringIO for comparison.

```python
from io import StringIO
import pandas as pd
from datacompy.core import Compare

data1 = """acct_id,dollar_amt,name,float_fld,date_fld
10000001234,123.45,George Maharis,14530.1555,2017-01-01
10000001235,0.45,Michael Bluth,1,2017-01-01
10000001236,1345,George Bluth,,2017-01-01
10000001237,123456,Bob Loblaw,345.12,2017-01-01
10000001238,1.05,Lucille Bluth,,2017-01-01
10000001238,1.05,Loose Seal Bluth,,2017-01-01
"""

data2 = """acct_id,dollar_amt,name,float_fld
10000001234,123.4,George Michael Bluth,14530.155
10000001235,0.45,Michael Bluth,
10000001236,1345,George Bluth,1
10000001237,123456,Robert Loblaw,345.12
10000001238,1.05,Loose Seal Bluth,111
"""

df1 = pd.read_csv(StringIO(data1))
df2 = pd.read_csv(StringIO(data2))
```

--------------------------------

### Set up Conda environment for DataComPy

Source: https://capitalone.github.io/datacompy/_sources/install.rst.txt

Create and activate a conda environment with Python 3.10, then configure conda-forge and install DataComPy. This method installs dependencies from Conda Forge.

```bash
conda create --name datacompy python=3.10 pip conda
```

```bash
source activate datacompy
```

```bash
conda config --add channels conda-forge
```

```bash
conda install datacompy
```

--------------------------------

### SnowflakeCompare Setup with Snowflake Table Names

Source: https://capitalone.github.io/datacompy/_sources/snowflake_usage.rst.txt

This snippet demonstrates setting up SnowflakeCompare using full Snowflake table names (db.schema.table_name). It assumes the tables have already been created and populated.

```python
df_1.write.mode("overwrite").save_as_table("toy_table_1")
df_2.write.mode("overwrite").save_as_table("toy_table_2")

compare = sp.SnowflakeCompare(
    session,
    f"{db}.{schema}.toy_table_1",
    f"{db}.{schema}.toy_table_2",
    #df1_name='original', # optional param for naming df1
    #df2_name='new' # optional param for naming df2
    join_columns=['acct_id'],
    rel_tol=1e-03,
    abs_tol=1e-04,
)
compare.matches(ignore_extra_columns=False)

# This method prints out a human-readable report summarizing and sampling differences
print(compare.report())

```

--------------------------------

### Example edgetest output

Source: https://capitalone.github.io/datacompy/developer_instructions.html

Sample output format returned after running the edgetest command.

```text
=============  ===============  ===================  =================
Environment    Passing tests    Upgraded packages    Package version
=============  ===============  ===================  =================
core           True             boto3                1.21.7
core           True             pandas               1.3.5
core           True             PyYAML               6.0
=============  ===============  ===================  =================
```

--------------------------------

### Example Jinja2 Template for Data Comparison Report

Source: https://capitalone.github.io/datacompy/template_guide.html

A basic Jinja2 template demonstrating how to display key comparison metrics like DataFrame shapes, column summaries, row summaries, and mismatched columns.

```jinja
# Data Comparison Report
=====================

## DataFrames
- {{ df1_name }}: {{ df1_shape[0] }} rows × {{ df1_shape[1] }} columns
- {{ df2_name }}: {{ df2_shape[0] }} rows × {{ df2_shape[1] }} columns

## Column Summary
- Common columns: {{ column_summary.common_columns }}
- Columns only in {{ df1_name }}: {{ column_summary.df1_unique }}
- Columns only in {{ df2_name }}: {{ column_summary.df2_unique }}

## Row Summary
- Rows with some columns unequal: {{ row_summary.unequal_rows }}
- Rows with all columns equal: {{ row_summary.equal_rows }}

{% if mismatch_stats %}
## Mismatched Columns
{% for col in mismatch_stats %}
- {{ col.column }}: {{ col.unequal_cnt }} mismatches
  - Match rate: {{ "%.2f"|format(col.match_rate * 100) }}%
{% endfor %}
{% endif %}


```

--------------------------------

### Setup PolarsCompare Object

Source: https://capitalone.github.io/datacompy/polars_usage.html

Instantiate `PolarsCompare` with two Polars DataFrames and specify join columns. Optional parameters like `abs_tol` and `rel_tol` can be set for numerical comparisons.

```python
compare = PolarsCompare(
    df1,
    df2,
    join_columns='acct_id',  #You can also specify a list of columns
    abs_tol=0.0001,
    rel_tol=0,
    df1_name='original',
    df2_name='new')

# OR

compare = PolarsCompare(df1, df2, join_columns=['acct_id', 'name'])
```

--------------------------------

### Upload to Real PyPI

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Upload the final distribution archives to the official PyPI repository. This makes your package available for public installation.

```bash
twine upload dist/*
```

--------------------------------

### SnowflakeCompare Setup with Snowpark DataFrames

Source: https://capitalone.github.io/datacompy/_sources/snowflake_usage.rst.txt

Use this snippet to set up SnowflakeCompare with Snowpark DataFrames. Ensure you have established a Snowflake session and prepared your data as Row objects.

```python
from snowflake.snowpark import Session
from snowflake.snowpark import Row
import datetime
import datacompy.snowflake as sp

connection_parameters = {
    ...
}
session = Session.builder.configs(connection_parameters).create()

data1 = [
Row(acct_id=10000001234, dollar_amt=123.45, name='George Maharis', float_fld=14530.1555,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001235, dollar_amt=0.45, name='Michael Bluth', float_fld=1.0,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001236, dollar_amt=1345.0, name='George Bluth', float_fld=None,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001237, dollar_amt=123456.0, name='Bob Loblaw', float_fld=345.12,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001239, dollar_amt=1.05, name='Lucille Bluth', float_fld=None,
    date_fld=datetime.date(2017, 1, 1)),
]

data2 = [
    Row(acct_id=10000001234, dollar_amt=123.4, name='George Michael Bluth', float_fld=14530.155),
    Row(acct_id=10000001235, dollar_amt=0.45, name='Michael Bluth', float_fld=None),
    Row(acct_id=None, dollar_amt=1345.0, name='George Bluth', float_fld=1.0),
    Row(acct_id=10000001237, dollar_amt=123456.0, name='Robert Loblaw', float_fld=345.12),
    Row(acct_id=10000001238, dollar_amt=1.05, name='Loose Seal Bluth', float_fld=111.0),
]

df_1 = session.createDataFrame(data1)
df_2 = session.createDataFrame(data2)

compare = sp.SnowflakeCompare(
    session,
    df_1,
    df_2,
    #df1_name='original', # optional param for naming df1
    #df2_name='new' # optional param for naming df2
    join_columns=['acct_id'],
    rel_tol=1e-03,
    abs_tol=1e-04,
)
compare.matches(ignore_extra_columns=False)

# This method prints out a human-readable report summarizing and sampling differences
print(compare.report())

```

--------------------------------

### Example Jinja2 Data Comparison Report Template

Source: https://capitalone.github.io/datacompy/_sources/template_guide.rst.txt

A basic Jinja2 template demonstrating how to format comparison metrics, DataFrame information, column summaries, and row summaries. It uses Jinja2 control structures like `{% if %}` and `{% for %}` to conditionally display mismatched column details.

```jinja
# Data Comparison Report
=====================

## DataFrames
- {{ df1_name }}: {{ df1_shape[0] }} rows × {{ df1_shape[1] }} columns
- {{ df2_name }}: {{ df2_shape[0] }} rows × {{ df2_shape[1] }} columns

## Column Summary
- Common columns: {{ column_summary.common_columns }}
- Columns only in {{ df1_name }}: {{ column_summary.df1_unique }}
- Columns only in {{ df2_name }}: {{ column_summary.df2_unique }}

## Row Summary
- Rows with some columns unequal: {{ row_summary.unequal_rows }}
- Rows with all columns equal: {{ row_summary.equal_rows }}

{% if mismatch_stats %}
## Mismatched Columns
{% for col in mismatch_stats %}
- {{ col.column }}: {{ col.unequal_cnt }} mismatches
  - Match rate: {{ "%.2f"|format(col.match_rate * 100) }}%
{% endfor %}
{% endif %}
```

--------------------------------

### Example of Duplicate Row Handling with Temporary IDs

Source: https://capitalone.github.io/datacompy/_sources/spark_usage.rst.txt

Illustrates how DataCompy internally generates temporary IDs to handle duplicate rows based on join columns, enabling precise matching before comparison.

```text
========== ================ ========
acct_id     name             temp_id
========== ================ בו
1           George Maharis   0
1           Michael Bluth    1
2           George Bluth     0
========== ================ בו
```

```text
========== ================ ======== 
acct_id     name             temp_id
========== ================ בו
1           George Maharis   0
1           Michael Bluth    1
1           Tony Wonder      2
2           George Bluth     0
========== ================ בו
```

--------------------------------

### datacompy.core.get_merged_columns

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Gets the merged columns from two DataFrames.

```APIDOC
## get_merged_columns()

### Description
Gets the merged columns from two DataFrames.

### Parameters
#### Path Parameters
- **df1** (sp.DataFrame) - Required - The first DataFrame.
- **df2** (sp.DataFrame) - Required - The second DataFrame.

### Returns
List of merged column names.
```

--------------------------------

### Upload to PyPI

Source: https://capitalone.github.io/datacompy/developer_instructions.html

Upload the generated distribution files to PyPI. Use the test PyPI first for verification.

```bash
# test pypi
twine upload --repository-url https://test.pypi.org/legacy/ dist/*

# real pypi
twine upload dist/*

```

--------------------------------

### GET /datacompy/polars/subset

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Checks if dataframe 2 is a subset of dataframe 1.

```APIDOC
## GET /datacompy/polars/subset

### Description
Return True if dataframe 2 is a subset of dataframe 1. Dataframe 2 is considered a subset if all of its columns are in dataframe 1, and all of its rows match rows in dataframe 1 for the shared columns.

### Response
#### Success Response (200)
- **is_subset** (bool) - True if dataframe 2 is a subset of dataframe 1.
```

--------------------------------

### Generate Distributions

Source: https://capitalone.github.io/datacompy/developer_instructions.html

Use the `build` module to create source and wheel distributions for your package.

```python
python -m build

```

--------------------------------

### Get Merged Columns

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Retrieves columns from an original dataframe that are present in a newly merged dataframe.

```APIDOC
## get_merged_columns

### Description
Get the columns from an original dataframe, in the new merged dataframe.

### Parameters
#### Path Parameters
- **original_df** (Pandas.DataFrame) - The original, pre-merge dataframe
- **merged_df** (Pandas.DataFrame) - Post-merge with another dataframe, with suffixes added in.
- **suffix** (str) - What suffix was used to distinguish when the original dataframe was overlapping with the other merged dataframe.

### Returns
List of strings representing the columns from the original dataframe found in the merged dataframe.
```

--------------------------------

### Generate Distribution Archives

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Use the 'build' module to create distribution archives (wheel and tar.gz) for your package. These will be placed in the 'dist' folder.

```bash
python -m build
```

--------------------------------

### Generate API Documentation

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Generate the API documentation using Sphinx from the root of the repository.

```bash
make sphinx
```

--------------------------------

### GET /datacompy/polars/sample_mismatch

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Retrieves a sub-dataframe containing identifying columns and the df1/df2 versions of a specific column for rows that do not match.

```APIDOC
## GET /datacompy/polars/sample_mismatch

### Description
Return sample mismatches for a specific column. Returns a sub-dataframe containing the identifying columns and the df1 and df2 versions of the column for rows that don't match.

### Parameters
#### Query Parameters
- **column** (str) - Required - The raw column name (without _df1 appended).
- **sample_count** (int) - Optional - The number of sample records to return. Defaults to 10.
- **for_display** (bool) - Optional - Whether this is used for display (overwrites column names).

### Response
#### Success Response (200)
- **result** (Polars.DataFrame | None) - A sample of the intersection dataframe or None if the column is not an intersecting column.
```

--------------------------------

### Initialize SnowflakeCompare with Snowflake Table Names

Source: https://capitalone.github.io/datacompy/snowflake_usage.html

Perform comparisons by providing the full database, schema, and table name strings instead of DataFrame objects.

```python
df_1.write.mode("overwrite").save_as_table("toy_table_1")
df_2.write.mode("overwrite").save_as_table("toy_table_2")

compare = sp.SnowflakeCompare(
    session,
    f"{db}.{schema}.toy_table_1",
    f"{db}.{schema}.toy_table_2",
    #df1_name='original', # optional param for naming df1
    #df2_name='new' # optional param for naming df2
    join_columns=['acct_id'],
    rel_tol=1e-03,
    abs_tol=1e-04,
)
compare.matches(ignore_extra_columns=False)

# This method prints out a human-readable report summarizing and sampling differences
print(compare.report())
```

--------------------------------

### Push Documentation to GitHub Pages

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Build and push the generated documentation to the gh-pages branch.

```bash
make ghpages
```

--------------------------------

### Generate documentation

Source: https://capitalone.github.io/datacompy/developer_instructions.html

Commands to build the API documentation using Sphinx and deploy it to the gh-pages branch.

```bash
make sphinx
```

```bash
make ghpages
```

--------------------------------

### Compare DataFrame Columns with DataComPy

Source: https://capitalone.github.io/datacompy/api/datacompy.spark.html

Use `columns_equal` to compare numeric columns with a relative tolerance or string columns ignoring spaces and case. Starting from version 0.18.0, this function returns a Column expression instead of a DataFrame.

```python
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([
    (1, 1.1, "ABC", None),
    (2, 2.0, "DEF ", 4.0),
    (3, None, "ghi", 5.0)
], ["id", "col1", "col2", "col3"])
# Compare numeric columns with tolerance
df = df.withColumn("col1_equals_col3", columns_equal(df, "col1", "col3", rel_tol=0.1))
# Compare string columns ignoring case and spaces
df = df.withColumn("col2_normalized", columns_equal(df, "col2", "col2", ignore_spaces=True, ignore_case=True))
```

--------------------------------

### Initialize SnowflakeCompare with Snowpark DataFrames

Source: https://capitalone.github.io/datacompy/snowflake_usage.html

Create a comparison object by passing a Snowpark session and two DataFrames. Ensure the session is configured correctly before creating the DataFrames.

```python
from snowflake.snowpark import Session
from snowflake.snowpark import Row
import datetime
import datacompy.snowflake as sp

connection_parameters = {
    ...
}
session = Session.builder.configs(connection_parameters).create()

data1 = [
Row(acct_id=10000001234, dollar_amt=123.45, name='George Maharis', float_fld=14530.1555,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001235, dollar_amt=0.45, name='Michael Bluth', float_fld=1.0,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001236, dollar_amt=1345.0, name='George Bluth', float_fld=None,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001237, dollar_amt=123456.0, name='Bob Loblaw', float_fld=345.12,
    date_fld=datetime.date(2017, 1, 1)),
Row(acct_id=10000001239, dollar_amt=1.05, name='Lucille Bluth', float_fld=None,
    date_fld=datetime.date(2017, 1, 1)),
]

data2 = [
    Row(acct_id=10000001234, dollar_amt=123.4, name='George Michael Bluth', float_fld=14530.155),
    Row(acct_id=10000001235, dollar_amt=0.45, name='Michael Bluth', float_fld=None),
    Row(acct_id=None, dollar_amt=1345.0, name='George Bluth', float_fld=1.0),
    Row(acct_id=10000001237, dollar_amt=123456.0, name='Robert Loblaw', float_fld=345.12),
    Row(acct_id=10000001238, dollar_amt=1.05, name='Loose Seal Bluth', float_fld=111.0),
]

df_1 = session.createDataFrame(data1)
df_2 = session.createDataFrame(data2)

compare = sp.SnowflakeCompare(
    session,
    df_1,
    df_2,
    #df1_name='original', # optional param for naming df1
    #df2_name='new' # optional param for naming df2
    join_columns=['acct_id'],
    rel_tol=1e-03,
    abs_tol=1e-04,
)
compare.matches(ignore_extra_columns=False)

# This method prints out a human-readable report summarizing and sampling differences
print(compare.report())
```

--------------------------------

### Upload to Test PyPI

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Upload the generated distribution archives to the Test PyPI repository for testing purposes before deploying to the real PyPI.

```bash
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
```

--------------------------------

### Initialize GitHub Pages Branch

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Create an orphaned branch for GitHub Pages, which is disconnected from the main codebase.

```bash
git checkout --orphan gh-pages
```

--------------------------------

### Generate Comparison Reports

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Generate a comparison report using the default template or a custom Jinja2 template file.

```python
>>> compare = datacompy.Compare(df1, df2, join_columns=['id'])
>>> report = compare.report()
>>> print(report)
```

```python
>>> # Create a custom template file (custom_report.j2)
>>> with open('custom_report.j2', 'w') as f:
...     f.write('''
...     Comparison Report
...     ================
...
...     DataFrames: {{ df1_name }} vs {{ df2_name }}
...
...     Shape Summary:
...     - {{ df1_name }}: {{ df1_shape[0] }} rows x {{ df1_shape[1] }} columns
...     - {{ df2_name }}: {{ df2_shape[0] }} rows x {{ df2_shape[1] }} columns
...
...     {% if mismatch_stats %}
...     Mismatched Columns ({{ mismatch_stats|length }}):
...     {% for col in mismatch_stats %}
...     - {{ col.column }} ({{ col.unequal_cnt }} mismatches)
...     {% endfor %}
...     {% else %}
...     No mismatches found in any columns!
...     {% endif %}
...     ''')
...
>>> # Generate report with custom template
>>> report = compare.report(template_path='custom_report.j2')
>>> print(report)
```

--------------------------------

### Using a Custom Template in DataComPy

Source: https://capitalone.github.io/datacompy/template_guide.html

Demonstrates how to generate a DataComPy report using a custom Jinja2 template by passing the template's file path to the `report()` method.

```python
from datacompy.core import Compare

compare = Compare(df1, df2, join_columns=['id'])

# Generate report with custom template
report = compare.report(template_path='path/to/your/template.j2')
print(report)


```

--------------------------------

### Compare PySpark DataFrames with SparkSQLCompare

Source: https://capitalone.github.io/datacompy/_sources/spark_usage.rst.txt

Initializes a SparkSQLCompare object to join and compare two DataFrames, then prints a summary report.

```python
from io import StringIO
import pandas as pd
from datacompy.spark.sql import SparkSQLCompare
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

data1 = """acct_id,dollar_amt,name,float_fld,date_fld
10000001234,123.45,George Maharis,14530.1555,2017-01-01
10000001235,0.45,Michael Bluth,1,2017-01-01
10000001236,1345,George Bluth,,2017-01-01
10000001237,123456,Bob Loblaw,345.12,2017-01-01
10000001239,1.05,Lucille Bluth,,2017-01-01
"""

data2 = """acct_id,dollar_amt,name,float_fld
10000001234,123.4,George Michael Bluth,14530.155
10000001235,0.45,Michael Bluth,
10000001236,1345,George Bluth,1
10000001237,123456,Robert Loblaw,345.12
10000001238,1.05,Loose Seal Bluth,111
"""

# SparkSQLCompare
df1 = spark.createDataFrame(pd.read_csv(StringIO(data1)))
df2 = spark.createDataFrame(pd.read_csv(StringIO(data2)))

compare = SparkSQLCompare(
    spark,
    df1,
    df2,
    join_columns='acct_id',  # You can also specify a list of columns
    abs_tol=0,  # Optional, defaults to 0
    rel_tol=0,  # Optional, defaults to 0
    df1_name='Original',  # Optional, defaults to 'df1'
    df2_name='New'  # Optional, defaults to 'df2'
)
compare.matches(ignore_extra_columns=False)
# False
# This method prints out a human-readable report summarizing and sampling differences
print(compare.report())
```

--------------------------------

### Generate Report with Custom Template in Python

Source: https://capitalone.github.io/datacompy/_sources/template_guide.rst.txt

Demonstrates how to generate a DataComPy report using a custom Jinja2 template. Ensure the `template_path` points to your `.j2` file. The generated report is then printed to the console.

```python
from datacompy.core import Compare

compare = Compare(df1, df2, join_columns=['id'])

# Generate report with custom template
report = compare.report(template_path='path/to/your/template.j2')
print(report)
```

--------------------------------

### SparkSQLCompare Methods

Source: https://capitalone.github.io/datacompy/api/datacompy.spark.html

Methods available on the SparkSQLCompare instance to retrieve comparison results.

```APIDOC
## Methods

### all_columns_match()
Returns True if all columns in df1 are in df2 and vice versa.

### all_mismatch(ignore_matching_cols=False)
Returns a DataFrame containing all rows with any columns that have a mismatch.

### all_rows_overlap()
Returns True if all rows in df1 are in df2 and vice versa.

### count_matching_rows()
Returns the number of matching rows as an int.

### matches(ignore_extra_columns=False)
Returns True or False if the dataframes match.

### report(sample_count=10, column_count=10, html_file=None, template_path=None)
Returns a string representation of a report.
```

--------------------------------

### SnowflakeCompare Class Initialization

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Initializes the comparison between two Snowpark DataFrames or Snowflake tables.

```APIDOC
## SnowflakeCompare Initialization

### Description
Initializes a comparison object to evaluate equality between two Snowpark DataFrames or Snowflake tables.

### Parameters
- **session** (snowflake.snowpark.session) - Required - Session with the required connection info.
- **df1** (str | DataFrame) - Required - First table name or Snowpark DataFrame.
- **df2** (str | DataFrame) - Required - Second table name or Snowpark DataFrame.
- **join_columns** (list | str) - Optional - Column(s) to join dataframes on.
- **abs_tol** (float | dict) - Optional - Absolute tolerance for value comparison.
- **rel_tol** (float | dict) - Optional - Relative tolerance for value comparison.
- **df1_name** (str) - Optional - Name for the first dataframe.
- **df2_name** (str) - Optional - Name for the second dataframe.
- **ignore_spaces** (bool) - Optional - Flag to strip whitespace from string columns.
```

--------------------------------

### Initialize Compare Object

Source: https://capitalone.github.io/datacompy/_sources/pandas_usage.rst.txt

Configure the Compare object using join columns or index to define how rows are matched between dataframes.

```python
compare = Compare(
    df1,
    df2,
    join_columns='acct_id',  #You can also specify a list of columns
    abs_tol=0.0001,
    rel_tol=0,
    df1_name='original',
    df2_name='new')

# OR

compare = Compare(df1, df2, join_columns=['acct_id', 'name'])

# OR

compare = Compare(df1, df2, on_index=True)
```

--------------------------------

### DataComPy Comparison Methods

Source: https://capitalone.github.io/datacompy/genindex.html

Common methods available across BaseCompare, Core, Polars, Snowflake, and SparkSQL comparison classes.

```APIDOC
## Comparison Methods

### Description
These methods are available across various DataComPy comparison classes to evaluate data differences.

### Methods
- **matches()**: Returns a boolean indicating if the datasets match.
- **report()**: Generates a text-based report of the comparison results.
- **sample_mismatch()**: Returns a sample of the rows that do not match.
- **subset()**: Checks if one dataset is a subset of the other.
```

--------------------------------

### Access Convenience Methods and Attributes

Source: https://capitalone.github.io/datacompy/spark_usage.html

Demonstrates accessing specific comparison results like intersecting rows or unique rows after running the comparison.

```python
print(compare.intersect_rows[['name_df1', 'name_df2', 'name_match']])
#          name_df1              name_df2  name_match
# 0  George Maharis  George Michael Bluth       False
# 1   Michael Bluth         Michael Bluth        True
# 2    George Bluth          George Bluth        True
# 3      Bob Loblaw         Robert Loblaw       False

print(compare.df1_unq_rows)
#    acct_id_df1  dollar_amt_df1       name_df1  float_fld_df1 date_fld_df1  _merge_left
```

--------------------------------

### Comparison Methods

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Methods available on the SnowflakeCompare object to analyze differences.

```APIDOC
## Comparison Methods

### all_columns_match()
Returns True if all columns in df1 are in df2 and vice versa.

### all_mismatch(ignore_matching_cols=False)
Returns a DataFrame containing all rows with any column mismatches.

### all_rows_overlap()
Returns True if all rows in df1 are in df2 and vice versa.

### count_matching_rows()
Returns the count of matching rows on overlapping fields.

### matches(ignore_extra_columns=False)
Returns True if the dataframes match, optionally ignoring extra columns.

### report(sample_count=10, column_count=10, html_file=None, template_path=None)
Returns a string representation of the comparison report.
```

--------------------------------

### Run Unit Tests

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Execute all unit tests defined in the 'tests' subfolder using pytest.

```bash
python -m pytest
```

--------------------------------

### columns_equal()

Source: https://capitalone.github.io/datacompy/api/datacompy.spark.html

Compares two columns for equality, with options for tolerance, ignoring spaces, and ignoring case. Returns a Column expression.

```APIDOC
## columns_equal()

### Description
Compares two columns for equality. Starting in version 0.18.0, this function returns a Column expression instead of a DataFrame.

### Method
N/A (Function within a DataFrame context)

### Endpoint
N/A

### Parameters
- **df** (DataFrame) - The input DataFrame.
- **col1_name** (str) - The name of the first column to compare.
- **col2_name** (str) - The name of the second column to compare.
- **rel_tol** (float, optional) - Relative tolerance for numeric comparisons. Defaults to 0.0.
- **abs_tol** (float, optional) - Absolute tolerance for numeric comparisons. Defaults to 0.0.
- **ignore_spaces** (bool, optional) - Whether to ignore leading/trailing spaces in string comparisons. Defaults to False.
- **ignore_case** (bool, optional) - Whether to ignore case in string comparisons. Defaults to False.

### Request Example
```python
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([
    (1, 1.1, "ABC", None),
    (2, 2.0, "DEF ", 4.0),
    (3, None, "ghi", 5.0)
], ["id", "col1", "col2", "col3"])

# Compare numeric columns with tolerance
df = df.withColumn("col1_equals_col3", columns_equal(df, "col1", "col3", rel_tol=0.1))

# Compare string columns ignoring case and spaces
df = df.withColumn("col2_normalized", columns_equal(df, "col2", "col2", ignore_spaces=True, ignore_case=True))
```

### Response
#### Success Response (Column Expression)
- Returns a PySpark Column expression that evaluates to True if the columns are equal based on the specified criteria, False otherwise.

#### Response Example
(A PySpark Column object representing the comparison result)
```

--------------------------------

### Method: report

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Generates a string representation of the comparison report.

```APIDOC
## Method report()

### Description
Return a string representation of a report. The representation can then be printed or saved to a file.

### Parameters
- **sample_count** (int) - Optional - The number of sample records to return. Defaults to 10.
- **column_count** (int) - Optional - The number of columns to display in the sample records output. Defaults to 10.
- **html_file** (str) - Optional - Path to save the report as an HTML file.
- **template_path** (str) - Optional - Path to a custom Jinja2 template.

### Response
- **Return Type** (str) - A string representation of the comparison report.
```

--------------------------------

### Report Generation

Source: https://capitalone.github.io/datacompy/api/datacompy.spark.html

Methods for generating and customizing comparison reports.

```APIDOC
## Report Generation

### Description
Generates a report of the comparison, which can be printed or saved to an HTML file using a custom Jinja2 template.

### Parameters
#### Query Parameters
- **sample_count** (int) - Optional - The number of sample records to return. Defaults to 10.
- **column_count** (int) - Optional - The number of columns to display in the sample records output. Defaults to 10.
- **html_file** (str) - Optional - HTML file name to save report output to.
- **template_path** (str) - Optional - Path to a custom Jinja2 template file.

### Response
- **Returns** (str) - The report, formatted according to the template.
```

--------------------------------

### Run Edgetest for Requirement Management

Source: https://capitalone.github.io/datacompy/_sources/developer_instructions.rst.txt

Execute edgetest to check and update project requirements. This command also updates pyproject.toml.

```bash
edgetest -c pyproject.toml --export
```

--------------------------------

### Accessing Comparison Results with Convenience Methods

Source: https://capitalone.github.io/datacompy/_sources/snowflake_usage.rst.txt

Use these methods on a comparison object to retrieve intersecting rows, unique rows, and column differences.

```python
compare.intersect_rows[['name_df1', 'name_df2', 'name_match']].show()
# --------------------------------------------------------
# |"NAME_DF1"      |"NAME_DF2"            |"NAME_MATCH"  |
# --------------------------------------------------------
# |George Maharis  |George Michael Bluth  |False         |
# |Michael Bluth   |Michael Bluth         |True          |
# |George Bluth    |George Bluth          |True          |
# |Bob Loblaw      |Robert Loblaw         |False         |
# --------------------------------------------------------

compare.df1_unq_rows.show()
# ---------------------------------------------------------------------------------------
# |"ACCT_ID_DF1"  |"DOLLAR_AMT_DF1"  |"NAME_DF1"     |"FLOAT_FLD_DF1"  |"DATE_FLD_DF1"  |
# ---------------------------------------------------------------------------------------
# |10000001239    |1.05              |Lucille Bluth  |NULL             |2017-01-01      |
# ---------------------------------------------------------------------------------------

compare.df2_unq_rows.show()
# -------------------------------------------------------------------------
# |"ACCT_ID_DF2"  |"DOLLAR_AMT_DF2"  |"NAME_DF2"        |"FLOAT_FLD_DF2"  |
# -------------------------------------------------------------------------
# |10000001238    |1.05              |Loose Seal Bluth  |111.0            |
# -------------------------------------------------------------------------

print(compare.intersect_columns())
# OrderedSet(['acct_id', 'dollar_amt', 'name', 'float_fld'])

print(compare.df1_unq_columns())
# OrderedSet(['date_fld'])

print(compare.df2_unq_columns())
# OrderedSet()
```

--------------------------------

### sample_mismatch

Source: https://capitalone.github.io/datacompy/api/datacompy.spark.html

Returns sample mismatches for a specific column.

```APIDOC
## sample_mismatch

### Description
Gets a sub-dataframe which contains the identifying columns, and df1 and df2 versions of the column for rows that don't match.

### Parameters
#### Query Parameters
- **column** (str) - Required - The raw column name.
- **sample_count** (int) - Optional - The number of sample records to return. Defaults to 10.
- **for_display** (bool) - Optional - Whether this is just going to be used for display.

### Response
- **Returns** (pyspark.sql.DataFrame) - A sample of the intersection dataframe.
```

--------------------------------

### SparkSQLCompare Class

Source: https://capitalone.github.io/datacompy/api/datacompy.spark.html

Initializes the comparison between two PySpark SQL DataFrames.

```APIDOC
## SparkSQLCompare

### Description
Comparison class to be used to compare whether two Spark SQL dataframes are equal.

### Parameters
- **spark_session** (pyspark.sql.SparkSession) - Required - A SparkSession to be used to execute Spark commands.
- **df1** (pyspark.sql.DataFrame) - Required - First dataframe to check.
- **df2** (pyspark.sql.DataFrame) - Required - Second dataframe to check.
- **join_columns** (list or str) - Optional - Column(s) to join dataframes on.
- **abs_tol** (float or dict) - Optional - Absolute tolerance between two values.
- **rel_tol** (float or dict) - Optional - Relative tolerance between two values.
- **df1_name** (str) - Optional - Name for the first dataframe.
- **df2_name** (str) - Optional - Name for the second dataframe.
- **ignore_spaces** (bool) - Optional - Flag to strip whitespace from string columns.
- **ignore_case** (bool) - Optional - Flag to ignore the case of string columns.
- **cast_column_names_lower** (bool) - Optional - Boolean indicator to cast column names to lower case.
```

--------------------------------

### datacompy.fugue.report

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Generates a comparison report for DataFrames.

```APIDOC
## fugue.report()

### Description
Generates a comparison report for DataFrames.

### Method
GET

### Endpoint
/websites/capitalone_github_io_datacompy

### Returns
String containing the comparison report.
```

--------------------------------

### Method: report()

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Generates a comparison report between two dataframes, optionally using a custom Jinja2 template.

```APIDOC
## report()

### Description
Generates a report summarizing the differences between two dataframes. The report can be saved to an HTML file or returned as a string.

### Parameters
- **sample_count** (int) - Optional - The number of sample records to return. Defaults to 10.
- **column_count** (int) - Optional - The number of columns to display in the sample records output. Defaults to 10.
- **html_file** (str) - Optional - HTML file name to save report output to.
- **template_path** (str) - Optional - Path to a custom Jinja2 template file.

### Response
- **str** - The report, formatted according to the template.
```

--------------------------------

### datacompy.core.Compare.df1

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Returns the original DataFrame.

```APIDOC
## Compare.df1

### Description
Returns the original DataFrame.

### Method
GET

### Endpoint
/websites/capitalone_github_io_datacompy

### Returns
Original DataFrame.
```

--------------------------------

### Compare String and Date Columns

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Compares a string column and a date column value-wise by attempting to convert the string column to a date column.

```APIDOC
## compare_string_and_date_columns

### Description
Compare a string column and date column, value-wise. This tries to convert a string column to a date column and compare that way.

### Parameters
#### Path Parameters
- **col_1** (Pandas.Series) - The first column to look at
- **col_2** (Pandas.Series) - The second column

### Returns
A series of Boolean values. True == the values match, False == the values don’t match.

### Return type
pandas.Series
```

--------------------------------

### Template Context Variables

Source: https://capitalone.github.io/datacompy/_sources/template_guide.rst.txt

Overview of the variables available within the Jinja2 template context for generating custom reports.

```APIDOC
## Template Context Variables

### Description
This section details the data structures passed to custom Jinja2 templates during report generation.

### Available Variables
- **column_summary** (dict) - Contains column statistics:
  - common_columns (list): Columns common to both dataframes
  - df1_unique (list): Columns unique to df1
  - df2_unique (list): Columns unique to df2
  - df1_name (string): Name of the first dataframe
  - df2_name (string): Name of the second dataframe

- **row_summary** (dict) - Contains row statistics:
  - match_columns (list): Columns used for matching
  - abs_tol (float): Absolute tolerance
  - rel_tol (float): Relative tolerance
  - common_rows (int): Number of common rows
  - df1_unique (int): Rows unique to df1
  - df2_unique (int): Rows unique to df2
  - unequal_rows (int): Number of rows with differences

- **column_comparison** (dict) - Contains column comparison stats:
  - unequal_columns (list): Columns with mismatches
  - equal_columns (list): Columns that match exactly
  - unequal_values (int): Count of unequal values

- **mismatch_stats** (dict) - Contains detailed mismatch data:
  - stats (list): Per-column mismatch statistics (column, match, mismatch, null_diff, total)
  - samples (list): Sample rows with mismatched values
  - has_samples (boolean): Indicates if samples are present
```

--------------------------------

### datacompy.base.BaseCompare.df1

Source: https://capitalone.github.io/datacompy/api/datacompy.html

Returns the original DataFrame.

```APIDOC
## BaseCompare.df1

### Description
Returns the original DataFrame.

### Method
GET

### Endpoint
/websites/capitalone_github_io_datacompy

### Returns
Original DataFrame.
```