### Install Project Requirements

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Installs all dependencies listed in the 'requirements.txt' file. Ensure your virtual environment is activated.

```console
pip3 install -r requirements.txt
```

--------------------------------

### Install Full Package from PyPI

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Use this command to install the complete DataProfiler package with all ML requirements from PyPI.

```console
pip install DataProfiler[ml]
```

--------------------------------

### Install Snappy for Parquet/Avro Profiling on macOS (Intel)

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Installs the snappy library using Homebrew and then installs the python-snappy package. Ensure you have Homebrew installed.

```console
brew install snappy && CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy
```

--------------------------------

### Install Snappy for Parquet/Avro Profiling on Linux

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Installs the snappy development library on Debian/Ubuntu-based Linux systems using apt-get.

```console
sudo apt-get -y install libsnappy-dev
```

--------------------------------

### Install Data Profiler Expectations

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/great_expectations/README.md

Install the data profiler expectation package and initialize Great Expectations.

```shell
cd examples/great_expectations/ # cd to great expectations examples
python3 -m venv venv # create venv
source venv/bin/activate # activate venv
pip install git+https://github.com/great-expectations/great_expectations.git@develop#egg=capitalone-dataprofiler-expectations&subdirectory=contrib/capitalone_dataprofiler_expectations/ # install data profiler expectation package
great_expectations init # init great expectations
python -m ipykernel install --name=venv # create a new kernel to use in jupyter notebook
```

--------------------------------

### Initialize ProfilerOptions with 'complete' Preset

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Initializes ProfilerOptions using the 'complete' preset for comprehensive profiling. No additional setup is required.

```python
options = ProfilerOptions(presets="complete")
```

--------------------------------

### Build and Install from Local Repository

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Builds the Python package distribution files (sdist, bdist, bdist_wheel) and then installs the wheel file locally. This is useful for development or testing local changes.

```console
python3 setup.py sdist bdist bdist_wheel
pip3 install dist/DataProfiler*-py3-none-any.whl
```

--------------------------------

### Install DataProfiler from GitHub

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Installs the DataProfiler package directly from its GitHub repository using pip. This ensures you get the latest development version.

```console
pip3 install git+https://github.com/capitalone/dataprofiler.git#egg=dataprofiler
```

--------------------------------

### Setup Data and Profiler

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/merge_profile_list.ipynb

Instantiates a Pandas DataFrame with dummy data and creates a list of two Profiler objects using this DataFrame.

```python
d = {'col1': [1, 2], 'col2': [3, 4]}
df = pd.DataFrame(data=d)

list_of_profiles = [dp.Profiler(df), dp.Profiler(df)]
```

```python
list_of_profiles
```

--------------------------------

### Install Test Requirements

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Installs the necessary packages for running tests, as defined in 'requirements-test.txt'. Ensure your virtual environment is activated.

```console
pip install -r requirements-test.txt
```

--------------------------------

### Import Data Profiler Libraries

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/column_name_labeler.ipynb

Initial setup to import required libraries and ensure the dataprofiler package is available in the path.

```python
import os
import sys
import json
from pprint import pprint

import pandas as pd

try:
    import dataprofiler as dp
except ImportError:
    sys.path.insert(0, '../..')
    import dataprofiler as dp
```

--------------------------------

### Install Sphinx Requirements and Project

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/README.md

Installs necessary Python packages for Sphinx documentation generation and the DataProfiler project itself. Run these commands from the root of the DataProfiler directory.

```bash
# install sphinx requirements
# install the requirements from the feature branch
pip install pandoc &&
pip install -r requirements.txt &&
pip install -r requirements-ml.txt && 
pip install -r requirements-reports.txt && 
pip install -r requirements-docs.txt  &&
pip install -e .
```

--------------------------------

### Install Snappy for Parquet/Avro Profiling on macOS (Apple Chip)

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Installs the snappy library using Homebrew for Apple Silicon Macs and then installs the python-snappy package. Ensure you have Homebrew installed.

```console
brew install snappy && CPPFLAGS="-I/opt/homebrew/include -L/opt/homebrew/lib" pip install python-snappy
```

--------------------------------

### Install Labeler Dependencies

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Installs dependencies specifically required for the labeler functionality, as listed in 'requirements-ml.txt'. Ensure your virtual environment is activated.

```console
pip3 install -r requirements-ml.txt
```

--------------------------------

### Install Reports Requirement

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Install the 'dataprofiler[reports]' package to enable reporting and graphing functionalities.

```console
pip install 'dataprofiler[reports]'
```

--------------------------------

### Initialize ProfilerOptions with 'lower_memory_sketching' Preset

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Initializes ProfilerOptions using the 'lower_memory_sketching' preset for reduced memory usage during profiling. No additional setup is required.

```python
options = ProfilerOptions(presets="lower_memory_sketching")
```

--------------------------------

### Import Libraries for Popmon Demo

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/popmon_dp_loader_example.ipynb

Imports necessary libraries for the dataprofiler and popmon demo. Ensure dataprofiler is installed.

```python
import os
import sys
try:
    sys.path.insert(0, '..')
    import dataprofiler as dp
except ImportError:
    import dataprofiler as dp
import pandas as pd
import popmon  # noqa
```

--------------------------------

### Execute Alert Script

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/avro/userdata1_intentionally_mislabled_file.txt

A simple JavaScript alert execution example.

```javascript
<pt>alert('hi')</pt>
```

--------------------------------

### Generate Sphinx Documentation

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/README.md

Navigates to the Sphinx documentation directory and executes the script to generate the documentation. This command should be run after all dependencies are installed.

```bash
cd _docs/docs
python update_documentation.py
```

--------------------------------

### Generate, Save, and Load Multiple Profiles

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/unstructured_profilers.ipynb

This example demonstrates generating profiles from multiple data sources, saving each profile, and then loading and merging them. This is useful for consolidating insights from distributed data.

```python
# Load a multiple files via the Data class
filenames = ["txt/sentence-3x.txt",
             "txt/sentence.txt"]
data_objects = []
for filename in filenames:
    data_objects.append(dp.Data(os.path.join(data_path, filename)))

print(data_objects)
# Generate and save profiles
for i in range(len(data_objects)):
    profile = dp.Profiler(data_objects[i])
    report = profile.report(report_options={"output_format":"compact"})
    print(json.dumps(report, indent=4))
    profile.save(filepath="data-"+str(i)+".pkl")


# Load profiles and add them together
profile = None
for i in range(len(data_objects)):
    if profile is None:
        profile = dp.Profiler.load("data-"+str(i)+".pkl")
    else:
        profile += dp.Profiler.load("data-"+str(i)+".pkl")


# Report the compact version of the profile
report = profile.report(report_options={"output_format":"compact"})
print(json.dumps(report, indent=4))
```

--------------------------------

### Install Slimmer Package from PyPI

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Install a lighter version of DataProfiler from PyPI if ML requirements are too strict. This version disables default sensitive data detection.

```console
pip install DataProfiler
```

--------------------------------

### Initialize ProfilerOptions with 'data_types' Preset

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Initializes ProfilerOptions using the 'data_types' preset, focusing on data type analysis. No additional setup is required.

```python
options = ProfilerOptions(presets="data_types")
```

--------------------------------

### Package Partitioning (Haskell)

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

A Haskell function signature for partitioning packages, demonstrating a concise type signature. This example is used to illustrate the concept of line length and naming conventions in different languages.

```Haskell
partitionPkgs :: NonEmpty (NonEmptySet Package) -> ([Prebuilt], [NonEmptySetBuildable])
```

--------------------------------

### Basic File Opening Comment

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

An example of a comment that states the obvious 'how' rather than the 'why'. This style is generally discouraged in production code.

```csharp
//open file  
File.Open(filename);
```

--------------------------------

### Import Graphs Module

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Import the necessary graphs module from the dataprofiler library. Ensure 'dataprofiler[reports]' is installed.

```python
from dataprofiler.reports import graphs
```

--------------------------------

### Guard Clause Pattern

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

An example demonstrating the use of multiple early returns to handle complex conditional logic and improve readability.

```python
if things_are_broken():
  print_error()
  return None

if input_is_invalid():
  print_error()
  return None

if easy_common_solution():
  return simple_answer()

intermediate = do_some_prep()
if prep_failed(intermediate):
  print_error()
  return None

if less_easy_solution(intermediate):
  return less_simple_answer(intermediate)

if weird_special_case(intermediate):
  return "cows"

perflog_event(FULL_ANSWER_REQUIRED)

more_intermediate = lots_more_prep(intermediate)
if invalid_state(more_intermediate):
  print_error()
  return None

if degenerate_result(more_intermediate):
  return 0

return complicated_answer(more_intermediate)
```

--------------------------------

### Example: Plot All Histograms from StructuredProfiler

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Demonstrates plotting histograms for all integer and float columns in a StructuredProfiler. Requires importing dataprofiler and graphs, creating a profiler, and calling plot_histograms.

```python
import dataprofiler as dp
from dataprofiler.reports import graphs


data = [[1, 'a', 1.0],
        [2, 'b', 2.2],
        [3, 'c', 3.5],
        [None, 'd', 10.0]]
profiler = dp.StructuredProfiler(data)

# This will plot all IntColumn and FloatColumn as histograms (The first and last column).
fig = graphs.plot_histograms(profiler)
fig.show()
```

--------------------------------

### Initialize ProfilerOptions with 'numeric_stats_disabled' Preset

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Initializes ProfilerOptions using the 'numeric_stats_disabled' preset, which disables numeric statistics. No additional setup is required.

```python
options = ProfilerOptions(presets="numeric_stats_disabled")
```

--------------------------------

### JavaScript Comment Example

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

Illustrates the difference between a useless comment explaining 'how' and a useful comment explaining 'why' a specific implementation detail was chosen. The 'why' provides valuable context for future maintainers.

```javascript
// Inverted because list comes down in reverse-alphabetical from employees view
 for(var n = employee.Count; n > 0; n--)  { ... }
```

--------------------------------

### File Opening Comment with Variable

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

Similar to the previous example, this comment explains a basic operation using a variable, illustrating a common pattern of redundant commenting.

```csharp
//open file  
x3.Open(name_variable);
```

--------------------------------

### Structured Difference Report Example

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Illustrates the structure of a difference report for structured data profiles, detailing statistics for global and data columns.

```python
{
    'global_stats': {
        'file_type': [str, str],
        'encoding': [str, str],
        'samples_used': int,
        'column_count': int,
        'row_count': int,
        'row_has_null_ratio': float,
        'row_is_null_ratio': float,
        'unique_row_ratio': float,
        'duplicate_row_count': int,
        'correlation_matrix': list[list[float]],
        'chi2_matrix': list[list[float]],
        'profile_schema': list[dict[str, int]]
    },
    'data_stats': [{
        'column_name': str,
        'data_type': [str, str],
        'data_label': [list[str], list[str], list[str]],
        'categorical': [str, str],
        'order': [str, str],
        'statistics': {
            'min': float,
            'max': float,
            'sum': float,
            'mean': float,
            'median': float,
            'mode': [list[float], list[float], list[float]],
            'median_absolute_deviation': float,
            'variance': float,
            'stddev': float,
            't-test': {
                't-statistic': float,
                'conservative': {'deg_of_free': int,
                                 'p-value': float},
                'welch': {'deg_of_free': float,
                          'p-value': float}},
            'psi': float,
            "chi2-test": {
                "chi2-statistic": float,
                "deg_of_free": int,
                "p-value": float
            },
            'unique_count': int,
            'unique_ratio': float,
            'categories': [list[str], list[str], list[str]],
            'gini_impurity': float,
            'unalikeability': float,
            'categorical_count': [dict[str, int], dict[str, int], dict[str, int]],
            'avg_predictions': [dict[str, float]],
            'label_representation': [dict[str, float]],
            'sample_size': int,
            'null_count': int,
            'null_types': [list[str], list[str], list[str]],
            'null_types_index': [dict[str, int], dict[str, int], dict[str, int]],
            'data_type_representation': [dict[str, float]]
        },
        "null_replication_metrics": {
            "class_prior": list[int],
            "class_sum": list[list[int]],
            "class_mean": list[list[int]]
        }
    }]
}
```

--------------------------------

### Example: Plot Histogram from Individual IntColumn Profiler

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Demonstrates plotting a histogram for an individual IntColumn profiler. This involves creating a pandas Series, updating an IntColumn profiler, and then plotting its histogram.

```python
import pandas as pd

from dataprofiler.profilers import IntColumn
from dataprofiler.reports import graphs


data = pd.Series([1, 2, 3], dtype=str)
profiler = IntColumn('example')
profiler.update(data)

# Plot the axes
ax = graphs.plot_col_histogram(profiler)

# get and show the figure of the plotted histogram
fig = ax.get_figure()
fig.show()
```

--------------------------------

### Import Libraries for Data Profiling and Great Expectations

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/great_expectations/ge_data_profiler_data_assistant.ipynb

Import necessary libraries including Great Expectations, DataProfiler, and specific modules for data assistants and metrics. Ensure these are installed before running.

```python
import os
from pathlib import Path

import great_expectations as ge
from great_expectations.cli.datasource import sanitize_yaml_and_save_datasource, check_if_datasource_name_exists
from great_expectations.core.batch import BatchRequest
from great_expectations.checkpoint import SimpleCheckpoint
from great_expectations.rule_based_profiler.data_assistant_result import (
    DataAssistantResult,
)
import dataprofiler as dp

from capitalone_dataprofiler_expectations.rule_based_profiler.data_assistant.data_profiler_structured_data_assistant import (
    DataProfilerStructuredDataAssistant,
)
from capitalone_dataprofiler_expectations.rule_based_profiler.data_assistant_result.data_profiler_structured_data_assistant_result import (
    DataProfilerStructuredDataAssistantResult,
)
import capitalone_dataprofiler_expectations.metrics.data_profiler_metrics
```

--------------------------------

### Verbose Logging Example

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

This snippet demonstrates excessive logging comments surrounding code, which can obscure the actual logic and is often refactored out in favor of dedicated logging frameworks.

```csharp
Console.WriteLine("About to instansiate HelperClass");
using (var writer = acquireFileWriterBySomeMeans()) {
  writer.WriteLine("About to instansiate HelperCLass");
}
// create an instance of HelperClass and store it in helperClass variable
var helperClass = new HelperClass();
Console.WriteLine("Created instance of HelperClass");
using (var writer = acquireFileWriterBySomeMeans()) {
  writer.WriteLine("Created instance of HelperCLass");
}
// ...
```

--------------------------------

### Set Up Great Expectations Datasource and Checkpoint

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/great_expectations/ge_data_profiler_data_assistant.ipynb

Configure a Great Expectations datasource for multiple batches using a PandasExecutionEngine and an InferredAssetFilesystemDataConnector. This setup is crucial for processing time-series data like taxi trip records.

```python
# Prepare Batch Request
context = ge.get_context()
datasource_config = f"""
    name: taxi_multi_batch_datasource
    class_name: Datasource
    module_name: great_expectations.datasource
    execution_engine:
        module_name: great_expectations.execution_engine
        class_name: PandasExecutionEngine
    data_connectors:
        inferred_data_connector_all_years:
            class_name: InferredAssetFilesystemDataConnector
            base_directory: ../data
            default_regex:
                group_names:
                    - data_asset_name
                    - year
                    - month
                pattern: (yellow_tripdata_sample)_(\d.*)-(\d.*)\.csv
"""
context.test_yaml_config(yaml_config=datasource_config)
sanitize_yaml_and_save_datasource(context, datasource_config, overwrite_existing=False)
batch_request = {
    "datasource_name": "taxi_multi_batch_datasource",
    "data_connector_name": "inferred_data_connector_all_years",
    "data_asset_name": "yellow_tripdata_sample",
    "data_connector_query": {"index": 0},
}

# Prepare a new expectation suite
context = ge.data_context.DataContext()
expectation_suite_name = "yellow_tripdata_sample"
expectation_suite = context.create_expectation_suite(
    expectation_suite_name=expectation_suite_name, overwrite_existing=True
)

validator = context.get_validator(
    batch_request=BatchRequest(**batch_request),
    expectation_suite_name=expectation_suite_name,
)

checkpoint_config = {
    "class_name": "SimpleCheckpoint",
    "validations": [
        {
            "batch_request": batch_request,
            "expectation_suite_name": expectation_suite_name,
        }
    ],
}

checkpoint = SimpleCheckpoint(
    f"{validator.active_batch_definition.data_asset_name}_{expectation_suite_name}",
    context,
    **checkpoint_config,
)
```

--------------------------------

### Initialize Model Components

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/column_name_labeler.ipynb

Instantiate the preprocessor, model, and postprocessor classes.

```python
# pre processor 
preprocessor = dp.labelers.data_processing.DirectPassPreprocessor()

# model
from dataprofiler.labelers.column_name_model import ColumnNameModel
model = ColumnNameModel(
    parameters=parameters,
    label_mapping=label_mapping,
)


# post processor
postprocessor = dp.labelers.data_processing.ColumnNameModelPostprocessor()
```

--------------------------------

### ProfilerOptions Presets

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Demonstrates how to initialize ProfilerOptions using predefined presets for different profiling configurations.

```APIDOC
## ProfilerOptions Presets

### Description
Initialize `ProfilerOptions` with predefined configurations for common profiling tasks.

### Method
Instantiation with preset string

### Endpoint
N/A (Code example)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
# Using the 'complete' preset
options = ProfilerOptions(presets="complete")

# Using the 'data_types' preset
options = ProfilerOptions(presets="data_types")

# Using the 'numeric_stats_disabled' preset
options = ProfilerOptions(presets="numeric_stats_disabled")

# Using the 'lower_memory_sketching' preset
options = ProfilerOptions(presets="lower_memory_sketching")
```

### Response
N/A (Code example)

#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/install.md

Sets up a Python 3 virtual environment named 'venv3' and activates it. This is recommended for managing project dependencies.

```console
python3 -m virtualenv --python=python3 venv3
source venv3/bin/activate
```

--------------------------------

### Build Data Docs and Open Validation Results

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/great_expectations/ge_data_profiler_data_assistant.ipynb

After running the checkpoint, build the data documentation to visualize the validation results. Then, open the specific validation result for review.

```python
context.build_data_docs()

validation_result_identifier = checkpoint_result.list_validation_result_identifiers()[0]
context.open_data_docs(resource_identifier=validation_result_identifier)
```

--------------------------------

### Python Function Docstring Example

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/README.md

Illustrates the reStructuredText format for Python function docstrings, including parameter types, return types, and an example block. Ensure parameters and return types are clearly defined.

```python
def format_data(self, predictions, verbose=False):
    """
    Formats word level labeling of the Unstructured Data Labeler as you want

    :param predictions: A 2D list of word level predictions/labeling
    :type predictions: Dict
    :param verbose: A flag to determine verbosity
    :type verbose: Bool
    :return: JSON structure containing specified formatted output
    :rtype: JSON

    :Example:
        Look at this test. Don't forget the double colons to make a code block::
            This is a codeblock
            Type example code here
    """
```

--------------------------------

### Save and Load Profile using Pickle

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Demonstrates how to save a profile to a .pkl file and load it back. This method is currently only supported for structured profiles.

```python
import json
from dataprofiler import Data, Profiler
```

--------------------------------

### Profile Structured Data

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/data_profiler_demo.ipynb

Example of profiling a CSV file and printing the resulting column data labels.

```python
# profile data and get labels for each column
data = dp.Data(os.path.join(data_folder, "csv/SchoolDataSmall.csv"))
profiler = dp.Profiler(data)
report = profiler.report()


print('\Label Predictions:\n' + '=' * 85)
print(get_structured_results(report))
```

--------------------------------

### Data Holder Example

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

A simple JavaScript object used to store data without internal manipulation logic.

```javascript
{"name": "Brian", "role": "admin"}
```

--------------------------------

### Initialize Output Directory and Data Loader

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/popmon_dp_loader_example.ipynb

Creates the target directory for reports if it does not exist and initializes the Data Profiler dataloader.

```python
if not os.path.exists(report_output_dir):
    os.makedirs(report_output_dir)
dp_dataframe = dp_dataloader(path)
```

--------------------------------

### Configure DataProfiler Options

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Demonstrates how to load data and configure various profiler options, such as disabling text analysis or enabling specific numeric statistics. Options can be toggled directly or set using a dictionary.

```python
import json
from dataprofiler import Data, Profiler, ProfilerOptions

# Load and profile a CSV file
data = Data("your_file.csv")
profile_options = ProfilerOptions()

#All of these are different examples of adjusting the profile options

# Options can be toggled directly like this:
profile_options.structured_options.text.is_enabled = False
profile_options.structured_options.text.vocab.is_enabled = True
profile_options.structured_options.int.variance.is_enabled = True
profile_options.structured_options.data_labeler.data_labeler_dirpath = \
    "Wheres/My/Datalabeler"
profile_options.structured_options.data_labeler.is_enabled = False

# A dictionary can be sent in to set the properties for all the options
profile_options.set({"structured_options.data_labeler.is_enabled": False, "min.is_enabled": False})

# Specific columns can be set/disabled/enabled in the same way
profile_options.structured_options.text.set({"max.is_enabled":True,
                                         "variance.is_enabled": True})

# numeric stats can be turned off/on entirely
profile_options.set({"is_numeric_stats_enabled": False})
profile_options.set({"int.is_numeric_stats_enabled": False})

profile = Profiler(data, options=profile_options)
```

--------------------------------

### Load an Existing Data Labeler

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/data_labeling.md

This snippet shows how to load a pre-trained structured data labeler from a specified directory. The `help()` method can be used to retrieve information about the labeler's parameters, inputs, and output formats.

```python
import dataprofiler as dp

data_labeler = dp.DataLabeler(
    labeler_type='structured', dirpath="/path/to/my/labeler")

# get information about the parameters/inputs/output formats for the DataLabeler
data_labeler.help()
```

--------------------------------

### Specify Data loading options

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/data_readers.ipynb

Shows how to instantiate the `Data` class with custom options to control data loading behavior. The specific allowed options depend on the data reader being used.

```python
import dataprofiler as dp

options = {...}  # allowed options are specified for each data reader.
data = dp.Data(data, options=options)
```

--------------------------------

### Get Category Mappings for CategoricalColumn

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/code.txt

Returns a mapping of categories to integers. It also includes mappings for null types at the end of the sequence.

```python
@property
def category_mappings(self):
        """
        Returns the mapping for categories
        :return:
        """
        num_categories = len(self._categories)
        category_mappings = OrderedDict(
            zip(self._categories, range(0, num_categories))
        )
        category_mappings.update(
            OrderedDict(
                zip(self.null_type, [num_categories] * len(self.null_type))
            )
        )
        return category_mappings
```

--------------------------------

### Save and Load Profiles

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Demonstrates saving and loading profiles using pickle or JSON formats.

```python
# Load a CSV file, with "," as the delimiter
data = Data("your_file.csv")

# Read data into profile
profile = Profiler(data)

# save structured profile to pkl file
profile.save(filepath="my_profile.pkl")

# load pkl file to structured profile
loaded_pkl_profile = dp.Profiler.load(filepath="my_profile.pkl")

print(json.dumps(loaded_pkl_profile.report(report_options={"output_format": "compact"}),
                                       indent=4))
```

```python
import json
from dataprofiler import Data, Profiler

# Load a CSV file, with "," as the delimiter
data = Data("your_file.csv")

# Read data into profile
profile = Profiler(data)

# save structured profile to json file
profile.save(filepath="my_profile.json", save_method="json")

# load json file to structured profile
loaded_json_profile = dp.Profiler.load(filepath="my_profile.json", load_method="json")

print(json.dumps(loaded_json_profile.report(report_options={"output_format": "compact"}),
                                       indent=4))
```

--------------------------------

### Check and Get Category Parameters

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/code.txt

Determines if a column is categorical and returns associated parameters. It checks the number of unique values against defined thresholds.

```python
@staticmethod
def _check_and_get_category_params(df_series):
        """
        Check whether column corresponds to category type and adds category
        parameters if it is.
        :param df_series: a given column
        :type df_series: pandas.core.series.Series
        :return: is_categorical_column and updated parameters for model if category
        type
        :rtype: (bool, dict)
        """

        is_categorical_column = False

        df_len = float(len(df_series))
        unique_elements = df_series.unique()
        num_unique = len(unique_elements)

        model = dict()

        # # TODO: Is this necessary if after FloatColumn check?
        # # Will cause error if trying to set categorical and is int index type
        #         # if FloatColumn._is_all_float_col(df_series):
        #         #     is_categorical_column = False

        if num_unique <= settings.MAXIMUM_UNIQUE_VALUES_TO_CLASSIFY_AS_CATEGORICAL:
            # If there are less than 20 unique values in total, detect that
            # column as categorical
            # NOTE: If sorted, can go before int columns
            model["categories"] = df_series.unique().tolist()
            is_categorical_column = True

        elif df_len and num_unique / df_len <= settings.CATEGORICAL_THRESHOLD_DEFAULT:
            # NOTE: If sorted, can go before int columns
            model["categories"] = df_series.unique().tolist()
            is_categorical_column = True

        return is_categorical_column, model
```

--------------------------------

### Importing Required Libraries

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/great_expectations/ge_expect_column_values_vs_profile_min.ipynb

Initializes the environment with necessary imports for Great Expectations, Data Profiler, and the custom expectation.

```python
import os

import pandas as pd
import numpy as np

# Great expectations imports
import great_expectations as ge
from capitalone_dataprofiler_expectations.expectations. \
    expect_column_values_to_be_equal_to_or_greater_than_profile_min \
    import ExpectColumnValuesToBeEqualToOrGreaterThanProfileMin
from great_expectations.self_check.util import build_pandas_validator_with_data

# Data Profiler import
import dataprofiler as dp
```

--------------------------------

### Get and Show Plot Figure

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Retrieves the figure object from an axes and displays it. Ensure the plotting environment is set up correctly before calling show().

```python
fig = ax.get_figure()
fig.show()
```

--------------------------------

### Import DataProfiler and TensorFlow

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/structured_profilers.ipynb

Imports the necessary libraries for data profiling and suppresses TensorFlow logging. Ensure DataProfiler is installed and accessible in your Python path.

```python
import os
import sys
import json

try:
    sys.path.insert(0, '..')
    import dataprofiler as dp
except ImportError:
    import dataprofiler as dp

data_path = "../dataprofiler/tests/data"

# remove extra tf loggin
import tensorflow as tf
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
```

--------------------------------

### Unstructured Difference Report Example

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Shows the structure of a difference report for unstructured data profiles, focusing on global statistics and data label statistics.

```python
{
    'global_stats': {
        'file_type': [str, str],
        'encoding': [str, str],
        'samples_used': int,
        'empty_line_count': int,
        'memory_size': float
    },
    'data_stats': {
        'data_label': {
            'entity_counts': {
                'word_level': dict[str, int],
                'true_char_level': dict[str, int],
                'postprocess_char_level': dict[str, int]
            },
            'entity_percentages': {
                'word_level': dict[str, float],
                'true_char_level': dict[str, float],
                'postprocess_char_level': dict[str, float]
            }
        },
        'statistics': {
            'vocab': [list[str], list[str], list[str]],
            'vocab_count': [dict[str, int], dict[str, int], dict[str, int]],
            'words': [list[str], list[str], list[str]],
            'word_count': [dict[str, int], dict[str, int], dict[str, int]]
        }
    }
}
```

--------------------------------

### Download Popmon Tutorial Data

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/popmon_dp_loader_example.ipynb

Downloads the 'flight_delays.csv.gz' dataset from popmon's resources and extracts it to 'flight_delays.csv'.

```python
import gzip
import shutil
popmon_tutorial_data = popmon.resources.data("flight_delays.csv.gz")
with gzip.open(popmon_tutorial_data, 'rb') as f_in:
    with open('./flight_delays.csv', 'wb') as f_out:
        shutil.copyfileobj(f_in, f_out)
```

--------------------------------

### Example of long variable names in a calculation

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

Illustrates a C++ expression where descriptive, long variable names are used for clarity in a complex calculation involving transformations.

```cpp
weapon_translation_worldspace = character_translation_worldspace + weapon_translation_characterspace.TransformedBy(characterspace_to_worldspace)
```

--------------------------------

### Data Class Initialization

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/data_readers.md

Demonstrates the basic usage of the Data class to automatically detect and load data from various file types.

```APIDOC
## Data Class Initialization

### Description
Initialize the Data class by passing a file path or URL. The class automatically detects the file type and loads the data.

### Method
Instantiate the `dp.Data` class

### Endpoint
N/A (Class instantiation)

### Request Example
```python
import dataprofiler as dp
data = dp.Data("your_file.csv")
```

### Response
An instance of a Data class (e.g., CSVData, JSONData) based on the input file type.
```

--------------------------------

### Load Data and Generate Profile for Saving

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/data_profiler_demo.ipynb

Loads data from a CSV file and generates a profile, preparing it for saving. Ensure 'dp' and 'os' are imported.

```python
# Load data
data = dp.Data(os.path.join(data_folder, "csv/diamonds.csv"))

# Generate a profile
profile = dp.Profiler(data)
```

--------------------------------

### Data Profiling with CSV

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/data_profiler_demo.ipynb

A concise example of profiling data from a CSV file. It involves creating a Data object and then a Profiler object, followed by generating a report.

```python
import dataprofiler as dp

data = dp.Data('my_data.csv')
profiler = dp.Profiler(data)
report = profiler.report(report_options={"output_format": "compact"})
```

--------------------------------

### Basic JavaScript Loop

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

A simple JavaScript for loop example. This snippet is used to contrast with a more informative comment that explains the reasoning behind the loop's implementation.

```javascript
for(var n = employee.Count; n > 0; n--)  { ... }
```

--------------------------------

### View Generated Documentation

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/README.md

Opens the generated Sphinx documentation in the default web browser. Ensure the documentation generation command has completed successfully before running this.

```bash
open index.html
```

--------------------------------

### Import Libraries for DataProfiler Expectation

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/great_expectations/ge_expect_column_value_confidence_for_data_label.ipynb

Imports necessary libraries including pandas, numpy, Great Expectations, and DataProfiler components. This setup is required before running any expectations.

```python
import os

import pandas as pd
import numpy as np

# Great expectations imports
import great_expectations as ge
from capitalone_dataprofiler_expectations.expectations. \
    expect_column_values_confidence_for_data_label_to_be_greater_than_or_equal_to_threshold \
    import ExpectColumnValuesConfidenceForDataLabelToBeGreaterThanOrEqualToThreshold
from great_expectations.self_check.util import build_pandas_validator_with_data

# Data Profiler imports
import dataprofiler as dp
```

--------------------------------

### Load and Predict with Library Model

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/column_name_labeler.ipynb

Quickly load a pre-existing labeler from the library and run a prediction.

```python
labeler_from_library = dp.DataLabeler.load_from_library('column_name_labeler')
```

```python
labeler_from_library.predict(data=["ssn"])
```

--------------------------------

### Example: Plot Specific Column Histogram from StructuredProfiler

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Shows how to plot a histogram for a specific column (by index) from a StructuredProfiler. This involves defining the column index and passing it to plot_histograms.

```python
columns_names = [0]
fig = graphs.plot_histograms(profiler, columns_names)
fig.show()
```

--------------------------------

### C# Basic Property Syntax

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

Demonstrates the concise C# syntax for a simple property with a getter and setter.

```csharp
public string Foo {get;set;}
```

--------------------------------

### Import Data Profiler Requirements

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/labeler.ipynb

Initializes the environment and imports necessary libraries including Data Profiler and TensorFlow.

```python
import os
import sys
import json
import pandas as pd

try:
    sys.path.insert(0, '..')
    import dataprofiler as dp
except ImportError:
    import dataprofiler as dp

# remove extra tf loggin
import tensorflow as tf
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
```

--------------------------------

### Glyph Advance Comment

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

A comment that explains a complex operation's purpose, providing context that might not be immediately obvious from the code itself. This is an example of a helpful 'what' comment when the 'why' is implied.

```c++
// Get the amount of glyph advance for the next character
end_bytes = Types::UTF8::NextUnsafe( c, 0, glyphVal );
```

--------------------------------

### Update Profile with New Data

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/structured_profilers.ipynb

Loads a text file, creates a profile, and then updates the profile with data from another text file. This demonstrates how to incrementally update a profile as new data becomes available, provided the schema matches.

```python
# Load and profile a CSV file
data = dp.Data(os.path.join(data_path, "csv/sparse-first-and-last-column-header-and-author.txt"))
profile = dp.Profiler(data)

# Update the profile with new data:
new_data = dp.Data(os.path.join(data_path, "csv/sparse-first-and-last-column-skip-header.txt"))
# new_data = dp.Data(os.path.join(data_path, "iris-utf-16.csv")) # will error due to schema mismatch
profile.update_profile(new_data)

# Take a peek at the data
print(data.data)
print(new_data.data)

# Report the compact version of the profile
report  = profile.report(report_options={"output_format":"compact"})
print(json.dumps(report, indent=4))
```

--------------------------------

### Identify Entities in Structured Data

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/data_labeling.md

Use this snippet to load data and a structured data labeler to make predictions and get labels for each cell in your dataset. Ensure you have a 'your_data.csv' file available.

```python
import dataprofiler as dp

# load data and data labeler
data = dp.Data("your_data.csv")
data_labeler = dp.DataLabeler(labeler_type='structured')

# make predictions and get labels per cell
predictions = data_labeler.predict(data)
```

--------------------------------

### Train a DataLabeler with a custom model

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/add_new_model_to_data_labeler.ipynb

Initializes a DataLabeler, configures preprocessor and postprocessor parameters, and trains the model on a provided dataset.

```python
# add the new LSTM model to the data labeler
data_labeler = dp.DataLabeler(labeler_type='structured', trainable=True)
data_labeler.set_model(model)

# set default label (one of the column names) to the preprocessor and postprocessor
processor_params = {'default_label': 'comment'}
data_labeler._preprocessor.set_params(**processor_params)
data_labeler._postprocessor.set_params(**processor_params)

# train the data labeler
save_dirpath="data_labeler_saved"
if not os.path.exists(save_dirpath):
    os.makedirs(save_dirpath)

epochs=2
data_labeler.fit(
    x=value_label_df[0], y=value_label_df[1], labels=labels, epochs=epochs)
if save_dirpath:
    data_labeler.save_to_disk(save_dirpath)
```

--------------------------------

### Import Libraries for DataProfiler Graph Pipeline

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/graph_data_demo.ipynb

Imports necessary libraries for using DataProfiler, including sys, os, pprint, and the dataprofiler library itself. Ensure dataprofiler is installed or accessible in the system path.

```python
import os
import sys
import pprint

try:
    sys.path.insert(0, '..')
    import dataprofiler as dp
except ImportError:
    import dataprofiler as dp

data_path = "../dataprofiler/tests/data"
```

--------------------------------

### Load Multiple Files and Merge Profiles

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/structured_profilers.ipynb

Load data from multiple files, generate individual profiles, save them, and then load and merge them into a single profile. This demonstrates a workflow for distributed profile generation and aggregation.

```python
# Load a multiple files via the Data class
filenames = ["csv/sparse-first-and-last-column-header-and-author.txt",
             "csv/sparse-first-and-last-column-skip-header.txt"]
data_objects = []
for filename in filenames:
    data_objects.append(dp.Data(os.path.join(data_path, filename)))


# Generate and save profiles
for i in range(len(data_objects)):
    profile = dp.Profiler(data_objects[i])
    profile.save(filepath="data-"+str(i)+".pkl")


# Load profiles and add them together
profile = None
for i in range(len(data_objects)):
    if profile is None:
        profile = dp.Profiler.load("data-"+str(i)+".pkl")
    else:
        profile += dp.Profiler.load("data-"+str(i)+".pkl")


# Report the compact version of the profile
report = profile.report(report_options={"output_format":"compact"})
print(json.dumps(report, indent=4))
```

--------------------------------

### Example of long variable names and complex expression

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/discussion_reddit.txt

Illustrates a scenario where extremely long variable names and a complex expression are used, potentially hindering readability. This is contrasted with a simpler mathematical equivalent.

```python
i_hate_readability_a_whole_lot = this_was_a_terrible_idea_but_yolo + simple_exprs_are_incomprehensible * i_hate_readability_a_whole_lot
```

```mathematica
x = w + b * x
```

--------------------------------

### Import dataprofiler library

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/data_readers.ipynb

This snippet shows how to import the dataprofiler library, handling potential import issues by adjusting the system path.

```python
import os
import sys

try:
    sys.path.insert(0, '..')
    import dataprofiler as dp
except ImportError:
    import dataprofiler as dp
```

--------------------------------

### Get Subclasses of BaseColumnProfile

Source: https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/tests/data/txt/code.txt

This static method returns a list of all available column profile subclasses, ordered by their intended profiling priority. This is used to determine the most appropriate profile type for a given column.

```python
@staticmethod
def _get_subclasses():
        """Returns all subclasses of the abstract base class."""
        # NOTE: these profilers are ordered. Test functionality if changed.
        return [
            NullColumn,
            TextColumn,
            DateTimeColumn,
            IpAddressColumn,
            OrderColumn,
            LatLongColumn,
            IntColumn,
            FloatColumn,
            CategoricalColumn,
        ]
```

--------------------------------

### Load DataLabeler with Components

Source: https://github.com/capitalone/dataprofiler/blob/main/examples/column_name_labeler.ipynb

Assemble the components into a DataLabeler and inspect model configuration.

```python
data_labeler = dp.DataLabeler.load_with_components(
    preprocessor=preprocessor,
    model=model,
    postprocessor=postprocessor,
)
data_labeler.model.help()
```

```python
pprint(data_labeler.label_mapping)
```

```python
pprint(data_labeler.model._parameters)
```

--------------------------------

### Example: Plot Missing Values Bar Chart for a Column

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Demonstrates plotting a bar chart of missing values for a specific column. This involves creating a pandas Series, profiling it as a StructuredColProfiler, and then calling plot_col_missing_values.

```python
import pandas as pd

from dataprofiler.profilers.profile_builder import StructuredColProfiler
from dataprofiler.reports import graphs


data = pd.Series([1, 2, 3, None, None, 4], name='example', dtype=str)
profiler = StructuredColProfiler(data)

# Plot the axes, can be a list of multiple columns
ax = graphs.plot_col_missing_values([profiler])
```

--------------------------------

### Example: Plot Missing Values Matrix from StructuredProfiler

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/graphs.md

Illustrates plotting a missing values matrix for a StructuredProfiler using pandas DataFrame. This requires creating a DataFrame with missing values, profiling it, and then calling plot_missing_values_matrix.

```python
import dataprofiler as dp
from dataprofiler.reports import graphs


data = pd.DataFrame(
    [[None, '', 1.0, '1/2/2021'],
     [3, None, 3.5, ''],
     [1, None, 1.0, '2/5/2020'],
     [None, 1, 10.0, '3/5/2020']],
    columns=['integer', 'str', 'float', 'datetime'],
    dtype=object
)
profiler = dp.StructuredProfiler(data)

# This will plot the missing values matrix for all columns.
fig = graphs.plot_missing_values_matrix(profiler)
fig.show()
```

--------------------------------

### Generate Profile Report with Different Output Formats

Source: https://github.com/capitalone/dataprofiler/blob/main/_docs/docs/source/profiler.md

Use the `report` method with `report_options` to specify the desired output format. Options include 'pretty', 'compact', 'serializable', and 'flat'.

```python
report  = profile.report(report_options={"output_format": "pretty"})
```

```python
report  = profile.report(report_options={"output_format": "compact"})
```

```python
report  = profile.report(report_options={"output_format": "serializable"})
```

```python
report  = profile.report(report_options={"output_format": "flat"})
```