### Example Project Folder Structure

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/reference/learn/integrations/dbt_tutorial.md

This illustrates the expected directory structure after cloning the repository and creating the initial subdirectories. It helps in verifying the setup.

```bash
dbt-tutorial
└── tutorial-dbt-gx-airflow/
    ├── airflow/
        └── .env
    ├── data/
    ├── great-expectations/
    ├── pgadmin-data/
    ├── ssh-keys/
    ├── airflow.Dockerfile
```

--------------------------------

### Full Example Code for Quick Start

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_create_a_data_context/_quick_start.md

This Python code snippet demonstrates the complete process for setting up a Great Expectations environment and obtaining a Data Context, including verification.

```python
import great_expectations

context = great_expectations.get_context()
print(type(context))
```

--------------------------------

### Full Example: Create Directory Data Asset

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_asset/_local_or_networked/_tab-directory_data_asset.md

A complete Python example demonstrating the creation of a Directory Data Asset for CSV files within a local or networked file system. Ensure prerequisites like Python installation and a preconfigured Data Context are met.

```python
import great_expectations as gx

# Assume data_context is a preconfigured DataContext object
# For example:
# data_context = gx.get_context()

# 1. Retrieve your Data Source
data_source_name = "my_filesystem_datasource"
my_data_source = data_context.sources.get_datasource(data_source_name)

# 2. Define your Data Asset's parameters
asset_name = "taxi_trips"
data_directory = "data/"

# 3. Add the Data Asset to your Data Source
# This example creates a Data Asset that can read .csv file data
my_data_source.add_directory_csv_asset(name=asset_name, data_directory=data_directory)

# To save the changes to the Data Context configuration:
# data_context.save_datasource(my_data_source)

```

--------------------------------

### Install DataProfiler from PyPI

Source: https://github.com/fivetran/great_expectations/blob/develop/contrib/capitalone_dataprofiler_expectations/README.md

Basic installation command for the DataProfiler package from PyPI.

```bash
pip install DataProfiler
```

--------------------------------

### Install GX with S3 Dependencies and Configure AWS CLI

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/components/reference_notebooks/_fluent_datasources_assumed_code_for_guides.ipynb

Install the Great Expectations library with S3 support and verify AWS CLI installation and credentials.

```bash
# Install GX with S3 dependencies
! pip install great_expectations[s3] # Installs Boto3 dependency
! great_expectations --version

# Configure credentials
## Prerequisite: Install the AWS CLI
### Confirm AWS CLI is installed; this will be used to configure AWS credentials.
! aws --version
### Confirm AWS credentials are configured properly
! aws sts get-caller-identity
```

--------------------------------

### Install Test Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the recommended extra dependencies for local development and testing.

```bash
pip install "'.[test]'"
```

--------------------------------

### Install PySpark and Apache Spark

Source: https://github.com/fivetran/great_expectations/blob/develop/CONTRIBUTING_CODE.md

Install the PySpark library, which also installs Apache Spark. This is necessary for using Spark for code testing.

```console
pip install pyspark
```

--------------------------------

### Install API Docs Dependencies

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/README.md

Installs dependencies required for building the API documentation.

```bash
pip install -r docs/sphinx_api_docs_source/requirements-dev-api-docs.txt
```

--------------------------------

### Install MySQL Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the MySQL extra dependencies.

```bash
pip install "'.[mysql]'"
```

--------------------------------

### Install Teradata Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Teradata extra dependencies.

```bash
pip install "'.[teradata]'"
```

--------------------------------

### Full Code Example

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/trigger_actions_based_on_results/create_a_checkpoint_with_actions.md

A complete example demonstrating the creation, saving, and retrieval of a Checkpoint with actions in Great Expectations.

```python
from great_expectations import DataContext
from great_expectations.checkpoint import Checkpoint

# Assume validation_definitions and actions are already created
validation_definitions = [
    {
        "name": "my_validation_definition",
        "batch_request": {
            "datasource_name": "my_datasource",
            "data_asset_name": "my_data_asset"
        },
        "expectations": [
            {
                "expectation_type": "expect_column_values_to_not_be_null",
                "kwargs": {"column": "my_column"}
            }
        ]
    }
]
actions = [
    {
        "name": "my_action",
        "action_class_name": "StoreValidationResultAction"
    }
]

# Create a Checkpoint
my_checkpoint = Checkpoint(
    name="my_checkpoint",
    validation_definitions=validation_definitions,
    actions=actions,
    result_format="Complete"
)

# Initialize Data Context
datacxt = DataContext(
    project_root_dir="/path/to/your/great_expectations/project"
)

# Add the Checkpoint to the Data Context
datacxt.add_checkpoint(checkpoint=my_checkpoint)

# Retrieve the Checkpoint from the Data Context
retrieved_checkpoint = datacxt.get_checkpoint(checkpoint_name="my_checkpoint")

# You can now use retrieved_checkpoint to run validations
# For example:
# results = retrieved_checkpoint.run()

```

--------------------------------

### Install BigQuery and GCP Extras

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use these commands to install the BigQuery and GCP extra dependencies.

```bash
pip install "'.[bigquery]'"
 pip install "'.[gcp]'"
```

--------------------------------

### Install Trino Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Trino extra dependencies.

```bash
pip install "'.[trino]'"
```

--------------------------------

### Install GX Cloud Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the dependencies needed for GX Cloud.

```bash
pip install "'.[cloud]'"
```

--------------------------------

### Install PostgreSQL Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the PostgreSQL extra dependencies.

```bash
pip install "'.[postgresql]'"
```

--------------------------------

### Check Java Installation

Source: https://github.com/fivetran/great_expectations/blob/develop/assets/docker/spark/README.md

Verify if Java is installed on your system. Download Java if it is not found.

```bash
java -version
```

--------------------------------

### Install Vertica Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Vertica extra dependencies.

```bash
pip install "'.[vertica]'"
```

--------------------------------

### Install Athena Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Athena extra dependencies.

```bash
pip install "'.[athena]'"
```

--------------------------------

### Install Arrow Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Arrow extra dependencies.

```bash
pip install "'.[arrow]'"
```

--------------------------------

### Full Code Example

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/run_validations/run_a_validation_definition.md

A complete Python code example demonstrating how to retrieve, run, and review a Validation Definition.

```python
import great_expectations as gx

# Assume context is a preconfigured Data Context
# context = gx.get_context()

# 1. Retrieve a Validation Definition
validation_definition_name = "my_validation_definition_name"
validation_definition = context.validation_definitions.get(validation_definition_name)

# 2. Define Batch Parameters
batch_parameters = {
    "datasource": "my_datasource",
    "data_asset": "my_data_asset",
    "batch_identifiers": {
        "date": "2023-01-01"
    }
}

# 3. Execute the Validation Definition's run() method
validation_results = validation_definition.run(**batch_parameters)

# 4. Review the Validation Results
print(validation_results.to_json_dict())
```

--------------------------------

### Full Example: Create and Review File Data Context

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_create_a_data_context/_file_data_context.md

A comprehensive example demonstrating the creation and review of a File Data Context in Python using Great Expectations.

```python
import great_expectations as gx

# Request the first File Data Context found, or create a new one
context = gx.get_context(mode="file")

# Print the Data Context configuration as a Python dictionary
print(context.to_json_dict())

# Request a specific File Data Context by providing a project root directory
# If a File Data Context exists in the specified folder, it will be instantiated and returned.
# If a File Data Context is not found in the specified folder, a new File Data Context will be created.
context_specific = gx.get_context(project_root_dir="/path/to/your/project", mode="file")

# Print the specific Data Context configuration
print(context_specific.to_json_dict())
```

--------------------------------

### Install Hive Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Hive extra dependencies.

```bash
pip install "'.[hive]'"
```

--------------------------------

### Full Code Example for Organizing Expectation Suites

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/define_expectations/organize_expectation_suites.md

A comprehensive example demonstrating the creation of an Expectation Suite, adding it to a Data Context, and incorporating expectations.

```python
from great_expectations.core import ExpectationSuite, Expectation

# Assume 'context' is a pre-configured DataContext
# Assume 'expectation' is a pre-defined Expectation object

# 1. Create an Expectation Suite
suite_name = "my_suite_name"
expectation_suite = ExpectationSuite(name=suite_name)

# 2. Add the Expectation Suite to your Data Context
context.add_expectation_suite(expectation_suite=expectation_suite)

# 3. Add the Expectation to the Expectation Suite
expectation_suite.add_expectation(expectation=expectation)

# 4. If you test and modify an Expectation after adding it, save modifications
#    (This assumes 'expectation' was modified after being added to the suite)
# expectation.save() # This method is deprecated, use context.save_expectation_suite instead
context.save_expectation_suite(expectation_suite=expectation_suite)

# To retrieve an Expectation Suite:
retrieved_expectation_suite = context.get_expectation_suite(expectation_suite_name=suite_name)

print(f"Expectation Suite '{suite_name}' created and added to Data Context.")
print(f"Number of expectations in suite: {len(retrieved_expectation_suite.expectations)}")

```

--------------------------------

### Install SQL Server Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the SQL Server extra dependencies.

```bash
pip install "'.[sql-server]'"
```

--------------------------------

### Install Excel Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Excel extra dependencies.

```bash
pip install "'.[excel]'"
```

--------------------------------

### Full Example of Ephemeral Data Context Creation

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_create_a_data_context/_ephemeral_data_context.md

This is a complete example demonstrating the creation and review of an Ephemeral Data Context.

```python
import great_expectations as gx

context = gx.get_context(mode="ephemeral")
print(context.to_json_dict())
```

--------------------------------

### Copy Environment Example File

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/README.md

Copies the example .env file to be used for local development environment variables. This file should not be committed.

```bash
cp docs/docusaurus/.env.example docs/docusaurus/.env
```

--------------------------------

### Install Pagerduty Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Pagerduty extra dependencies.

```bash
pip install "'.[pagerduty]'"
```

--------------------------------

### Install GX Core

Source: https://github.com/fivetran/great_expectations/blob/develop/README.md

Install GX Core within a Python virtual environment using pip.

```bash
pip install great_expectations
```

--------------------------------

### Install S3 Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the S3 extra dependencies.

```bash
pip install "'.[s3]'"
```

--------------------------------

### Start PostgreSQL Docker Container

Source: https://github.com/fivetran/great_expectations/blob/develop/CONTRIBUTING_CODE.md

Use this command to start the PostgreSQL container for local testing. Ensure you are in the correct directory.

```sh
docker-compose up -d
```

--------------------------------

### Install DataProfiler with Full Dependencies

Source: https://github.com/fivetran/great_expectations/blob/develop/contrib/capitalone_dataprofiler_expectations/README.md

Use this command to install the full DataProfiler package from PyPI, including all optional dependencies.

```bash
pip install DataProfiler[full]
```

--------------------------------

### Install great_expectations_contrib

Source: https://github.com/fivetran/great_expectations/blob/develop/contrib/cli/README.md

Install the tool using pip. This command is typically run from the root of the Great Expectations repository.

```bash
pip install -e .
```

--------------------------------

### Install Azure and Azure Secrets Extras

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use these commands to install the Azure and Azure Secrets extra dependencies.

```bash
pip install "'.[azure]'"
 pip install "'.[azure_secrets]'"
```

--------------------------------

### PostgreSQL Database Does Not Exist Error Example

Source: https://github.com/fivetran/great_expectations/blob/develop/CONTRIBUTING_CODE.md

Example of a SQLAlchemy OperationalError indicating that the required database does not exist, which can occur during initial setup.

```console
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  database "test_ci" does not exist
(Background on this error at: http://sqlalche.me/e/e3q8)
```

--------------------------------

### Install boto3 Library

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_install_additional_dependencies/_amazon_s3.md

Install the boto3 library, which allows Python to interact with AWS services. This is a prerequisite for Great Expectations to work with S3.

```bash
python -m pip install boto3
```

```bash
python3 -m pip install boto3
```

--------------------------------

### Create a Quick Start Data Context

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/create_a_data_context.md

Use this snippet to quickly set up a Data Context for initial exploration and learning. It initializes a project with default configurations.

```python
import great_expectations as gx

context = gx.get_context("quickstart")
```

--------------------------------

### Create Expectation Suite and Expectations

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/reference/learn/data_quality_use_cases/unstructured_data.md

Get or create an Expectation Suite and add Expectations to validate metrics extracted from PDFs. This example uses `ExpectColumnValuesToBeBetween`.

```python
expectation_suite_name = "pdf_expectation_suite"
expectation_suite = context.add_expectation_suite(expectation_suite_name=expectation_suite_name)

expectation_suite.add_expectation(expectation_configuration=ExpectationConfiguration(
    expectation_type="expect_column_values_to_be_between",
    kwargs={
        "column": "page_count",
        "min_value": 1,
        "max_value": 100,
    },
    comment_id="my_pandas_datasource--pdf_asset--pdf_batch--pdf_expectation_suite",
))

context.save_expectation_suite(expectation_suite=expectation_suite)

```

--------------------------------

### Full Code Example for Configuring Metadata Stores

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/configure_project_settings/configure_metadata_stores/configure_metadata_stores.md

A comprehensive example demonstrating the complete process of configuring metadata stores, from loading the Data Context to re-initializing it with updated settings.

```python
# full code example
```

--------------------------------

### Clone dbt Tutorial Repo and Set Up Directories

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/reference/learn/integrations/dbt_tutorial.md

Clone the dbt tutorial GitHub repository and create necessary subdirectories for data, Great Expectations, and SSH keys. This sets up the initial project structure.

```bash
git clone https://github.com/greatexpectationslabs/dbt-tutorial.git
cd dbt-tutorial/tutorial-dbt-gx-airflow
mkdir data
mkdir great-expectations
mkdir ssh-keys
```

--------------------------------

### Start Development Server (after initial build)

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/README.md

Starts the development server without copying and building prior versions. Use this for subsequent runs after the initial `invoke docs`.

```console
invoke docs --start
```

--------------------------------

### Connect to Azure Blob Storage with Pandas

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/components/reference_notebooks/_fluent_datasources_assumed_code_for_guides.ipynb

Instantiate a Great Expectations Data Context and configure a Datasource for Azure Blob Storage using Pandas. This example assumes ABS dependencies are installed and credentials are configured.

```python
import great_expectations as gx

context = gx.get_context()
```

```python
datasource_name = "MyDatasource"
datasource = context.datasources.add_pandas_abs(
    name=datasource_name, azure_options={"account_url": "", "credentials": ""}
)
```

```python
csv_file_name_as_regex = r"taxi_data\.csv"
data_asset = datasource.add_csv_asset(
    asset_name="MyTaxiDataAsset", batching_regex=csv_file_name_as_regex
)
```

--------------------------------

### Full Spark Filesystem Data Source Example (Python)

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_source/_local_or_networked/_local_or_networked.md

Complete example code for setting up a Spark Filesystem Data Source in Great Expectations, including defining parameters and adding the data source.

```python
```python title="Python" name="docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_source/_local_or_networked/_spark.py - full example"
```
```

--------------------------------

### Install GX with S3 Dependencies

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/components/reference_notebooks/_fluent_datasources_assumed_code_for_guides.ipynb

Install the Great Expectations library with the necessary AWS S3 dependencies. This also confirms the installed version.

```python
# Install GX with S3 dependencies
! pip install great_expectations[s3] # Installs Boto3 dependency
! great_expectations --version
```

--------------------------------

### Full Pandas Filesystem Data Source Example (Python)

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_source/_local_or_networked/_local_or_networked.md

Complete example code for setting up a pandas Filesystem Data Source in Great Expectations, including defining parameters and adding the data source.

```python
```python title="Python" name="docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_source/_local_or_networked/_pandas.py - full example"
```
```

--------------------------------

### Verify GX Installation with Python

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/components/setup/_gx_verify_installation.md

Run this Python code to check if Great Expectations is installed correctly. If successful, it prints the installed version number.

```python
import great_expectations

print(great_expectations.__version__)
```

--------------------------------

### Full Code Example for Data Docs Configuration

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/configure_project_settings/configure_data_docs/configure_data_docs.md

A complete Python code example demonstrating the configuration and management of Data Docs sites, including adding a site, building documentation, and opening it for viewing.

```Python
from great_expectations.data_context.types.base import DataDocsSiteNames
from great_expectations.checkpoint.actions import UpdateDataDocsAction

# Assume 'context' is an initialized DataContext object
# context = great_expectations.get_context()

# 1. Define a configuration dictionary for your new Data Docs site.
# GX writes Data Doc sites to a directory specified by the `base_directory` key of the configuration dictionary.
# Configuring other keys of the dictionary is not supported, and they may be removed in a future release.

# 2. Add your configuration to your Data Context.
# All Data Docs sites have a unique name within a Data Context. Once your Data Docs site configuration has been defined, add it to the Data Context by updating the value of `site_name` in the following to something more descriptive and then execute the code::
context.add_data_docs_site(
    site_name="my_docs",
    site_config={
        "class_name": "SiteNames.LOCAL_QUANTUM",
        "store_backend": {
            "class_name": "UnchangedStoreBackend",
            "base_directory": "uncommitted/data_docs/my_docs",
        },
        "data_docs_sites": {
            "uncommitted/data_docs/my_docs": {},
        },
    }
)

# 3. Optional. Build your Data Docs sites manually.
# You can manually build a Data Docs site by executing the following code:
context.build_data_docs(site_names=["my_docs"])

# 4. Optional. Automate Data Docs site updates with Checkpoint Actions.
# You can automate the creation and update of Data Docs sites by including the `UpdateDataDocsAction` in your Checkpoints. This Action will automatically trigger a Data Docs site build whenever the Checkpoint it is included in completes its `run()` method.
# Assume 'checkpoint' is an initialized Checkpoint object
# checkpoint.add_action(UpdateDataDocsAction(action_list=[{ "name": "update_data_docs", "site_names": ["my_docs"] }]))

# 5. Optional. View your Data Docs.
# Once your Data Docs have been created, you can view them with:
context.open_data_docs(site_names=["my_docs"])

```

--------------------------------

### Full Sample Code for Creating Batch Definitions

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/sql_data/_create_a_batch_definition/_create_a_batch_definition.md

This is a comprehensive example demonstrating the creation of both full table and partitioned batch definitions.

```python
import great_expectations as gx

# Assume context is a preconfigured DataContext
# context = gx.get_context()

# Replace with your actual datasource and asset names
datasource_name = "my_datasource"
asset_name = "my_asset"

# --- Full Table Batch Definition ---
batch_definition_full = gx.BatchDefinition(
    name="my_full_table_batch_definition",
    datasource_name=datasource_name,
    data_asset_name=asset_name,
    batch_spec_passthrough={
        "splitter_method": None
    },
)

# --- Partitioned Batch Definition (Daily) ---
date_column = "my_date_column"
batch_definition_partitioned = gx.BatchDefinition(
    name="my_daily_batch_definition",
    datasource_name=datasource_name,
    data_asset_name=asset_name,
    batch_spec_passthrough={
        "splitter_method": "split_on_date_strings",
        "date_strings_format": "%Y-%m-%d",
        "date_strings_partition_by_size": "D",
        "date_strings_column_name": date_column,
    },
)

# To add these batch definitions to your Data Asset, you would typically
# use the DataContext's add_batch_definition method or similar.
# Example (conceptual - actual implementation might vary based on GX version/API):
# context.add_batch_definition(batch_definition_full)
# context.add_batch_definition(batch_definition_partitioned)

print("Batch definitions created (but not yet added to Data Context).")
print(f"Full Table Batch Definition: {batch_definition_full.id}")
print(f"Partitioned Batch Definition: {batch_definition_partitioned.id}")

```

--------------------------------

### Full Example: Directory Data Asset for Azure Blob Storage

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_asset/_abs/_tab-directory_data_asset.md

A complete example demonstrating the creation of a Directory Data Asset for Azure Blob Storage, including prerequisites and configuration.

```python
import great_expectations as gx

# Assuming gx.get_context() has been called and a Data Context is configured
context = gx.get_context()

# Retrieve the existing Spark Filesystem Data Source
datasource = context.get_datasource("my_datasource_name")

# Add a Directory Data Asset for CSV files in Azure Blob Storage
datasource.add_directory_csv_asset(
    name="taxi_yellow_tripdata_samples",
    abs_container="my_abs_container",
    abs_name_starts_with="data/taxi_yellow_tripdata_samples/",
    data_directory="taxi_yellow_tripdata_samples/",
    abs_recursive_file_discovery=True
)

# Save the updated Data Context configuration
context.save_expectation_suite("my_expectation_suite_name")

```

--------------------------------

### Install PostgreSQL Development Headers

Source: https://github.com/fivetran/great_expectations/blob/develop/CONTRIBUTING_CODE.md

Installs the `libpq-dev` package required for installing the `psycopg2-binary` Python package when using Amazon Redshift or PostgreSQL. This is an optional step.

```sh
sudo apt-get install -y libpq-dev
```

--------------------------------

### Install UnixODBC Development Headers

Source: https://github.com/fivetran/great_expectations/blob/develop/CONTRIBUTING_CODE.md

Installs the `unixodbc-dev` package required for installing the `pyodbc` Python package when using Microsoft SQL Server or Dremio. This is an optional step.

```sh
sudo apt-get install -y unixodbc-dev
```

--------------------------------

### Verify AWS CLI Version

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_install_additional_dependencies/_amazon_s3.md

Run this command to check if the AWS Command Line Interface is installed and to see its version. If it's not installed or outdated, you'll need to install or update it.

```bash
aws --version
```

--------------------------------

### Setup BigQuery Performance Test Data

Source: https://github.com/fivetran/great_expectations/blob/develop/CONTRIBUTING_CODE.md

Run this command to set up the necessary data in BigQuery for performance testing. Replace <YOUR_GCP_PROJECT> with your actual GCP project ID.

```bash
GE_TEST_BIGQUERY_PEFORMANCE_DATASET=<YOUR_GCP_PROJECT> tests/performance/setup_bigquery_tables_for_performance_test.sh
```

--------------------------------

### Full Example: Create a Tabular Data Asset

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_asset/_local_or_networked/_tab-file_data_asset.md

This is a complete example demonstrating the creation of a Data Asset for CSV files from a filesystem. Ensure you have a preconfigured Data Context and Data Source.

```Python
from great_expectations.data_context import DataContext

context = DataContext(
    project_root_dir="/path/to/your/great_expectations/project",
    context_root_dir="/path/to/your/great_expectations/project/great_expectations",
)

datasource = context.get_datasource("data_source_name")
asset_name = "taxi_csv_files"
datasource.add_csv_asset(name=asset_name)
```

--------------------------------

### Install Snowflake Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Snowflake extra dependencies.

```bash
pip install "'.[snowflake]'"
```

--------------------------------

### Full Code Example

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/define_expectations/_retrieve_a_batch_of_test_data/_from_pandas_default.md

A complete example demonstrating how to retrieve a batch of test data using the pandas_default Data Source, including defining the file path, reading the data, and verifying its contents.

```python
csv_file_path = "../data/titanic.csv"
batch_of_data = context.sources.pandas_default.read_csv(csv_file_path)
print(batch_of_data.head())
```

--------------------------------

### Install Redshift Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Redshift extra dependencies.

```bash
pip install "'.[redshift]'"
```

--------------------------------

### Running the Build Gallery Script

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/expectation_gallery/1-the-build_gallery.py-script.md

Commands to activate a virtual environment, navigate to the script directory, and execute the build_gallery.py script. Useful for setting up and running local tests.

```bash
source venv/bin/activate

cd assets/scripts

python ./build_gallery.py [OPTIONS] [ARGS]
```

--------------------------------

### Full Code Example - Python

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/dataframes/dataframes.md

A complete example demonstrating the creation of a DataFrame Data Asset. This includes retrieving the Data Source, defining the Data Asset name, and adding the Data Asset.

```python
datasource = context.get_datasource("my_datasource_name")
data_asset_name = "my_new_dataframe_data_asset_name"
datasource.add_dataframe_asset(name=data_asset_name)
```

--------------------------------

### Install Dremio Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Dremio extra dependencies.

```bash
pip install "'.[dremio]'"
```

--------------------------------

### Class Diagram for DataSource and Batch Test Setup

Source: https://github.com/fivetran/great_expectations/blob/develop/tests/integration/data_sources_and_expectations/README.md

Illustrates the relationships between DataSourceTestConfig, BatchTestSetup, and their specific implementations for PostgreSQL and Snowflake.

```mermaid
classDiagram
    class DataSourceTestConfig
    class BatchTestSetup

    DataSourceTestConfig <|-- PostgreSQLDatasourceTestConfig
    DataSourceTestConfig <|-- SnowflakeDatasourceTestConfig
    BatchTestSetup <|-- PostgresBatchTestSetup
    BatchTestSetup <|-- SnowflakeBatchTestSetup

    <<Abstract>> DataSourceTestConfig
    <<Abstract>> BatchTestSetup

    DataSourceTestConfig : +str label
    DataSourceTestConfig : +str pytest_marks
    DataSourceTestConfig : +dict column_types
    DataSourceTestConfig : +create_batch_setup(data) BatchTestSetup

    BatchTestSetup  : +DataSourceTestConfig config
    BatchTestSetup  : +dict data
    BatchTestSetup  : +setup()
    BatchTestSetup  : +teardown()
    BatchTestSetup  : +make_batch() Batch

    DataSourceTestConfig  --> BatchTestSetup: creates
```

--------------------------------

### Install ClickHouse Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the ClickHouse extra dependencies.

```bash
pip install "'.[clickhouse]'"
```

--------------------------------

### Full Sample Code for Creating Data Assets

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/sql_data/_create_a_data_asset/_create_a_data_asset.md

This is a complete Python script that includes prerequisites, data source retrieval, and the creation of both Table and Query Data Assets.

```python
from ruamel.yaml import YAML
from great_expectations import get_context

context = get_context()

data_source_name = "my_sql_datasource"

data_source = context.sources.get(data_source_name)

# Add a Table Asset
data_source.add_table_asset(
    name="my_table_asset",
    table_name="my_table"
)

# Add a Query Asset
data_source.add_query_asset(
    name="my_query_asset",
    query="SELECT * FROM my_table WHERE column_a > 5"
)

print(data_source.assets)
```

--------------------------------

### Build Static Documentation Site

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/README.md

Generates a static version of the documentation site into the 'build' directory, suitable for deployment on any static hosting service.

```console
invoke docs --build
```

--------------------------------

### Install Spark Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the Spark extra dependencies.

```bash
pip install "'.[spark]'"
```

--------------------------------

### Install Great Expectations

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/components/reference_notebooks/_fluent_datasources_assumed_code_for_guides.ipynb

Installs the Great Expectations library using pip.

```bash
! pip install great_expectations
```

--------------------------------

### Install Great Expectations S3 Support

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_install_additional_dependencies/_amazon_s3.md

Install the optional dependencies for Great Expectations to enable support for AWS S3. This command installs GX Core and the boto3 library if they are not already present.

```bash
python -m pip install 'great_expectations[s3]'
```

--------------------------------

### Filtering Build Gallery Script Backends and Expectations

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/expectation_gallery/1-the-build_gallery.py-script.md

Example of how to specify a subset of backends and a specific Expectation for faster local testing. This is useful for quick iteration during development.

```bash
python ./build_gallery.py --backends "pandas, sqlite" expectation_name
```

--------------------------------

### Install Trino Dependencies

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_install_additional_dependencies/_sql_dialect_installation_commands.md

Use this command to install Great Expectations with support for Trino.

```bash
pip install 'great_expectations[trino]'
```

--------------------------------

### Install Databricks Dependencies

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/set_up_a_gx_environment/_install_additional_dependencies/_sql_dialect_installation_commands.md

Use this command to install Great Expectations with support for Databricks.

```bash
pip install 'great_expectations[databricks]'
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/README.md

Installs development dependencies for Great Expectations, including PySpark.

```bash
pip install -c constraints-dev.txt -e ".[test]"
pip install pyspark
```

--------------------------------

### Create and Run a Checkpoint

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/introduction/try_gx.md

Set up and execute a Checkpoint to validate your data against the defined Validation Definition. The `.describe()` method provides a summary of the validation results.

```python
checkpoint_yaml = "name: my_checkpoint

validation_definitions:
  - my_validation_definition
"
context.add_checkpoint(**yaml.safe_load(checkpoint_yaml))
results = context.run_checkpoint(checkpoint_name="my_checkpoint")
print(results.describe())

```

--------------------------------

### Full Code Example for Creating Expectations

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/define_expectations/create_an_expectation.md

This comprehensive Python example illustrates the complete process of defining an Expectation Suite with dynamic expectations and then executing a Checkpoint with runtime parameters. It includes dataset creation, expectation definition, and checkpoint execution.

```python
import great_expectations as gx
import pandas as pd

# 1. Create a DataContext
context = gx.get_context()

# 2. Create a Pandas DataFrame
data = {
    "passenger_count": [1, 2, 3, 4, 5],
    "fare": [10.5, 20.0, 15.75, 30.0, 25.5],
}
df = pd.DataFrame(data)

# 3. Create an Expectation Suite
expectation_suite_name = "my_dynamic_suite"
suite = context.create_expectation_suite(expectation_suite_name=expectation_suite_name)

# 4. Add dynamic expectations
suite.add_expectation(
    expectation_type="expect_column_max_to_be_between",
    kwargs={
        "column": "passenger_count",
        "min_value": {"$PARAMETER": "min_passenger_count"},
        "max_value": {"$PARAMETER": "max_passenger_count"},
    },
)
suite.add_expectation(
    expectation_type="expect_column_max_to_be_between",
    kwargs={
        "column": "fare",
        "min_value": {"$PARAMETER": "min_fare"},
        "max_value": {"$PARAMETER": "max_fare"},
    },
)

# Save the expectation suite
context.save_expectation_suite(suite)

# 5. Create a Checkpoint
checkpoint_name = "my_dynamic_checkpoint"
checkpoint_config = {
    "name": checkpoint_name,
    "config_version": 1.0,
    "class_name": "Checkpoint",
    "validations": [
        {
            "expectation_suite_name": expectation_suite_name,
            "batch_request": {
                "datasource_name": "my_datasource",
                "data_asset_name": "my_data_asset",
                "batch_spec_passthrough": {"reader_method": "python_set"},
                "batch_parameters": {"data": df},
            },
        }
    ],
}
context.add_checkpoint(**checkpoint_config)

# 6. Define runtime parameters
runtime_params = {
    "min_passenger_count": 1,
    "max_passenger_count": 6,
    "min_fare": 0,
    "max_fare": 100,
}

# 7. Run the Checkpoint with runtime parameters
checkpoint_result = context.run_checkpoint(
    checkpoint_name=checkpoint_name,
    expectation_suite_name=expectation_suite_name,
    expectation_parameters=runtime_params,
)

# Print the result (optional)
print(checkpoint_result)

```

--------------------------------

### Running build_gallery.py for a Specific Backend

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/expectation_gallery/1-the-build_gallery.py-script.md

This snippet shows how to run the `build_gallery.py` script for a single backend, redirecting output to a file and then filtering for errors and tracebacks.

```bash
python ./build_gallery.py --backends "SOME-BACKEND" 2>&1 | tee output--build_gallery.txt
grep -o "ERROR - (.*" output--build_gallery.txt | sort > testing-error-messages.txt
touch gallery-tracebacks.txt

# Show testing errors
cat testing-error-messages.txt

# Show gallery tracebacks
cat gallery-tracebacks.txt
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/reference/learn/data_quality_use_cases/unstructured_data.md

Installs essential Python libraries for the tutorial, including pandas for data manipulation, datasets for data handling, pdf2image for PDF processing, pytesseract for Tesseract integration, and great_expectations for data quality validation.

```bash
pip install pandas
pip install datasets
pip install pdf2image
pip install pytesseract
pip install great_expectations
```

--------------------------------

### Full Sample Code for Whole Directory Batch Definition

Source: https://github.com/fivetran/great_expectations/blob/develop/docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_batch_definition/_tab-directory_batch_definition.md

This is a complete Python example for creating and using a whole directory batch definition. It covers the entire process from definition to verification.

```python
```python title="Full sample code" name="docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_batch_definition/_examples/_directory_whole_directory.py - full_example"
```
```

--------------------------------

### Install AWS Secrets Extra

Source: https://github.com/fivetran/great_expectations/blob/develop/reqs/README.md

Use this command to install the AWS Secrets extra dependencies.

```bash
pip install "'.[aws_secrets]'"
```

--------------------------------

### Install and Run Algolia Upload Scripts

Source: https://github.com/fivetran/great_expectations/blob/develop/assets/scripts/AlgoliaScripts/README.md

Install project dependencies using npm and then run the scripts to upload expectations and packages data to Algolia. Ensure you are in the AlgoliaScripts directory.

```bash
cd AlgoliaScripts
npm install

// Upload expectations to algolia
node upload-expec

// Upload packages data to algolia
node upload-packages
```