### Build and Install DataProfiler from Local Repo Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Build the source distribution and wheel for DataProfiler, then install the local wheel file. This is used when installing from the repository. ```console python3 setup.py sdist bdist bdist_wheel pip3 install dist/DataProfiler*-py3-none-any.whl ``` -------------------------------- ### Install Requirements from File Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install project dependencies listed in the 'requirements.txt' file within an activated virtual environment. ```console pip3 install -r requirements.txt ``` -------------------------------- ### Install snappy-dev on Linux Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install the 'snappy-dev' package using apt-get. This is required for profiling parquet/avro datasets. ```console sudo apt-get -y install libsnappy-dev ``` -------------------------------- ### Install Popmon Source: https://capitalone.github.io/DataProfiler/popmon_dp_loader_example.html Use this command to install the popmon library. Imports are included in the next step. ```bash pip3 install popmon ``` -------------------------------- ### Install python-snappy on MacOS (Intel) Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install the 'snappy' library using Homebrew and then install the 'python-snappy' package. This is required for profiling parquet/avro datasets. ```console brew install snappy && CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy ``` -------------------------------- ### Install python-snappy on MacOS (Apple Chip) Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install the 'snappy' library using Homebrew and then install the 'python-snappy' package. This is required for profiling parquet/avro datasets. ```console brew install snappy && CPPFLAGS="-I/opt/homebrew/include -L/opt/homebrew/lib" pip install python-snappy ``` -------------------------------- ### Install Test Requirements Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install the necessary dependencies for running tests, as specified in 'requirements-test.txt'. ```console pip3 install -r requirements-test.txt ``` -------------------------------- ### Install DataProfiler Reports Source: https://capitalone.github.io/DataProfiler/_sources/graphs.rst.txt Install the DataProfiler library with the reports extra to enable plotting functionalities. ```console pip install 'dataprofiler[reports]' ``` -------------------------------- ### Install DataProfiler from GitHub Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install the DataProfiler package directly from its GitHub repository using pip. ```console pip3 install git+https://github.com/capitalone/dataprofiler.git#egg=dataprofiler ``` -------------------------------- ### Install Full DataProfiler Package Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Use this command to install the complete DataProfiler package with all ML dependencies from PyPI. ```console pip install DataProfiler[ml] ``` -------------------------------- ### Install ML Requirements from File Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install specific machine learning dependencies listed in the 'requirements-ml.txt' file within an activated virtual environment. ```console pip3 install -r requirements-ml.txt ``` -------------------------------- ### Install Test Requirements Source: https://capitalone.github.io/DataProfiler/install.html Installs the necessary packages for running tests, as defined in requirements-test.txt. ```bash pip install -r requirements-test.txt ``` -------------------------------- ### Import Libraries for DataLabeler Example Source: https://capitalone.github.io/DataProfiler/add_new_model_to_data_labeler.html Import necessary libraries including os, sys, json, pandas, and the DataProfiler library for the example. ```python import os import sys import json import pandas as pd sys.path.insert(0, '..') import dataprofiler as dp ``` -------------------------------- ### Setup DataProfiler Environment Source: https://capitalone.github.io/DataProfiler/unstructured_profiler_example.html Imports necessary libraries and sets up the DataProfiler environment, including handling TensorFlow logging. ```python import os import sys import json try: sys.path.insert(0, '..') import dataprofiler as dp except ImportError: import dataprofiler as dp data_path = "../dataprofiler/tests/data" # remove extra tf loggin import tensorflow as tf tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR) ``` -------------------------------- ### Initialize ProfilerOptions with Presets Source: https://capitalone.github.io/DataProfiler/_sources/profiler.rst.txt Shows how to initialize ProfilerOptions using predefined presets like 'complete'. Use presets for quick setup of common option configurations. ```python options = ProfilerOptions(presets="complete") ``` -------------------------------- ### Install DataProfiler with ML Support Source: https://capitalone.github.io/DataProfiler/index.html Install the full DataProfiler package from PyPI, including ML capabilities for sensitive data detection and entity recognition. ```bash pip install DataProfiler[ml] ``` -------------------------------- ### Create and Activate Virtual Environment Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Set up a Python 3 virtual environment named 'venv3' and activate it. This is a prerequisite for building and installing from source. ```console python3 -m virtualenv --python=python3 venv3 source venv3/bin/activate ``` -------------------------------- ### Setup Data and Profiler Source: https://capitalone.github.io/DataProfiler/merge_profile_list.html Instantiates a Pandas DataFrame with dummy data and then creates a list containing two separate DataProfiler instances, each initialized with the same DataFrame. ```python d = {'col1': [1, 2], 'col2': [3, 4]} df = pd.DataFrame(data=d) list_of_profiles = [dp.Profiler(df), dp.Profiler(df)] ``` -------------------------------- ### Building a Labeler Pipeline from Scratch Source: https://capitalone.github.io/DataProfiler/labeler.html This example shows how to construct a DataProfiler labeler by manually setting its components: model, preprocessor, and postprocessor. It also demonstrates how to set parameters for these components and check the pipeline's compatibility. ```python import random from dataprofiler.labelers.character_level_cnn_model import \ CharacterLevelCnnModel from dataprofiler.labelers.data_processing import \ StructCharPreprocessor, StructCharPostprocessor model = CharacterLevelCnnModel({"PAD":0, "UNKNOWN":1, "Test_Label":2}) preprocessor = StructCharPreprocessor() postprocessor = StructCharPostprocessor() labeler = dp.DataLabeler(labeler_type='structured') labeler.set_preprocessor(preprocessor) labeler.set_model(model) labeler.set_postprocessor(postprocessor) # check for basic compatibility between the processors and the model labeler.check_pipeline() # Optionally set the parameters parameters={ 'preprocessor':{ 'max_length': 100, }, 'model':{ 'max_length': 100, }, 'postprocessor':{ 'random_state': random.Random(1) } } labeler.set_params(parameters) labeler.help() ``` -------------------------------- ### Example of Ascending Order Profile Update Source: https://capitalone.github.io/DataProfiler/_sources/profiler.rst.txt Demonstrates how to update data batches to maintain an ascending order for order profiling. ```python batch_1 = [0, 1, 2] batch_2 = [3, 4, 5] ``` -------------------------------- ### Example: Plot All Histograms from StructuredProfiler Source: https://capitalone.github.io/DataProfiler/graphs.html This example demonstrates plotting histograms for all numeric columns in a StructuredProfiler. It also shows how to plot histograms for a specific column by name. ```python import dataprofiler as dp from dataprofiler.reports import graphs data = [[1, 'a', 1.0], [2, 'b', 2.2], [3, 'c', 3.5], [None, 'd', 10.0]] profiler = dp.StructuredProfiler(data) # This will plot all IntColumn and FloatColumn as histograms (The first and last column). fig = graphs.plot_histograms(profiler) fig.show() # This will only plot the specified column, 0. columns_names = [0] fig = graphs.plot_histograms(profiler, columns_names) fig.show() ``` -------------------------------- ### Setup Python 3 Virtual Environment Source: https://capitalone.github.io/DataProfiler/install.html Creates and activates a Python 3 virtual environment named 'venv3' for isolated project dependencies. ```bash python3 -m pip install virtualenv ``` ```bash python3 -m virtualenv --python=python3 venv3 source venv3/bin/activate ``` -------------------------------- ### Adjusting Profiler Options in Python Source: https://capitalone.github.io/DataProfiler/profiler.html Demonstrates how to load data, instantiate ProfilerOptions, and modify various profiling settings directly or via a dictionary. Includes examples for enabling/disabling specific features and columns. ```python import json from dataprofiler import Data, Profiler, ProfilerOptions # Load and profile a CSV file data = Data("your_file.csv") profile_options = ProfilerOptions() #All of these are different examples of adjusting the profile options # Options can be toggled directly like this: profile_options.structured_options.text.is_enabled = False profile_options.structured_options.text.vocab.is_enabled = True profile_options.structured_options.int.variance.is_enabled = True profile_options.structured_options.data_labeler.data_labeler_dirpath = \ "Wheres/My/Datalabeler" profile_options.structured_options.data_labeler.is_enabled = False # A dictionary can be sent in to set the properties for all the options profile_options.set({"structured_options.data_labeler.is_enabled": False, "min.is_enabled": False}) # Specific columns can be set/disabled/enabled in the same way profile_options.structured_options.text.set({"max.is_enabled":True, "variance.is_enabled": True}) # numeric stats can be turned off/on entirely profile_options.set({"is_numeric_stats_enabled": False}) profile_options.set({"int.is_numeric_stats_enabled": False}) profile = Profiler(data, options=profile_options) ``` -------------------------------- ### Example: Plot All Histograms from StructuredProfiler Source: https://capitalone.github.io/DataProfiler/_sources/graphs.rst.txt Demonstrates plotting histograms for all numeric columns in a StructuredProfiler. Requires `dataprofiler` and `dataprofiler.reports`. ```python import dataprofiler as dp from dataprofiler.reports import graphs data = [[1, 'a', 1.0], [2, 'b', 2.2], [3, 'c', 3.5], [None, 'd', 10.0]] profiler = dp.StructuredProfiler(data) # This will plot all IntColumn and FloatColumn as histograms (The first and last column). fig = graphs.plot_histograms(profiler) fig.show() ``` -------------------------------- ### Example: Plot Histogram from Individual IntColumn Profile Source: https://capitalone.github.io/DataProfiler/_sources/graphs.rst.txt Demonstrates plotting a histogram from an individual `IntColumn` profile. Requires `pandas` and `dataprofiler.reports`. ```python import pandas as pd from dataprofiler.profilers import IntColumn from dataprofiler.reports import graphs data = pd.Series([1, 2, 3], dtype=str) profiler = IntColumn('example') profiler.update(data) # Plot the axes ax = graphs.plot_col_histogram(profiler) # get and show the figure of the plotted histogram fig = ax.get_figure() fig.show() ``` -------------------------------- ### Generate, Save, and Merge Multiple Profiles Source: https://capitalone.github.io/DataProfiler/unstructured_profiler_example.html This example demonstrates generating profiles from multiple files, saving each profile, and then loading and merging them. This is useful for distributed profiling or processing large datasets in chunks. ```python import os import json import dataprofiler as dp data_path = "/path/to/your/data" # Load multiple files via the Data class filenames = ["txt/sentence-3x.txt", "txt/sentence.txt"] data_objects = [] for filename in filenames: data_objects.append(dp.Data(os.path.join(data_path, filename))) print(data_objects) # Generate and save profiles for i in range(len(data_objects)): profile = dp.Profiler(data_objects[i]) report = profile.report(report_options={"output_format":"compact"}) print(json.dumps(report, indent=4)) profile.save(filepath="data-"+str(i)+".pkl") # Load profiles and add them together profile = None for i in range(len(data_objects)): if profile is None: profile = dp.Profiler.load("data-"+str(i)+".pkl") else: profile += dp.Profiler.load("data-"+str(i)+".pkl") # Report the compact version of the merged profile report = profile.report(report_options={"output_format":"compact"}) print(json.dumps(report, indent=4)) ``` -------------------------------- ### Example: Plot Missing Values Matrix from StructuredProfiler Source: https://capitalone.github.io/DataProfiler/_sources/graphs.rst.txt Demonstrates plotting a missing values matrix for all columns in a StructuredProfiler. Requires `dataprofiler` and `dataprofiler.reports`. ```python import dataprofiler as dp from dataprofiler.reports import graphs data = pd.DataFrame( [[None, '', 1.0, '1/2/2021'], [3, None, 3.5, ''], [1, None, 1.0, '2/5/2020'], [None, 1, 10.0, '3/5/2020']], columns=['integer', 'str', 'float', 'datetime'], dtype=object ) profiler = dp.StructuredProfiler(data) # This will plot the missing values matrix for all columns. fig = graphs.plot_missing_values_matrix(profiler) fig.show() ``` -------------------------------- ### Get Help for RegexModel Parameters Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Display help information describing the alterable parameters of the RegexModel. This method provides guidance on available configurations. ```python RegexModel._help() ``` -------------------------------- ### Initialize RegexModel with Patterns and Encapsulators Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Initialize the RegexModel by providing a mapping of labels to their regex patterns and defining encapsulator patterns for word boundaries. This setup is crucial for the model's prediction capabilities. ```python regex_patterns = { “LABEL_1”: [ “LABEL_1_pattern_1”, “LABEL_1_pattern_2”, … ], “LABEL_2”: [ > “LABEL_2_pattern_1”, “LABEL_2_pattern_2”, … } encapsulators = { ‘start’: r’(?: { range: { 'start': 1, 'end':2 }, list: [1,2,3] } } ``` -------------------------------- ### Unstructured Difference Report Example Source: https://capitalone.github.io/DataProfiler/profiler.html This is an example of an unstructured difference report, which may contain different statistics compared to the structured report, particularly for text-based data. ```python { 'global_stats': { 'file_type': [str, str], 'encoding': [str, str], 'samples_used': int, 'empty_line_count': int, 'memory_size': float }, 'data_stats': { 'data_label': { 'entity_counts': { 'word_level': dict[str, int], 'true_char_level': dict[str, int], 'postprocess_char_level': dict[str, int] }, 'entity_percentages': { 'word_level': dict[str, float], 'true_char_level': dict[str, float], 'postprocess_char_level': dict[str, float] } }, 'statistics': { 'vocab': [list[str], list[str], list[str]], 'vocab_count': [dict[str, int], dict[str, int], dict[str, int]], 'words': [list[str], list[str], list[str]], 'word_count': [dict[str, int], dict[str, int], dict[str, int]] } } } ``` -------------------------------- ### Structured Difference Report Example Source: https://capitalone.github.io/DataProfiler/profiler.html This is an example of a structured difference report generated by the .diff() method, detailing differences in global statistics and individual data columns. ```python { 'global_stats': { 'file_type': [str, str], 'encoding': [str, str], 'samples_used': int, 'column_count': int, 'row_count': int, 'row_has_null_ratio': float, 'row_is_null_ratio': float, 'unique_row_ratio': float, 'duplicate_row_count': int, 'correlation_matrix': list[list[float]], 'chi2_matrix': list[list[float]], 'profile_schema': list[dict[str, int]] }, 'data_stats': [{ 'column_name': str, 'data_type': [str, str], 'data_label': [list[str], list[str], list[str]], 'categorical': [str, str], 'order': [str, str], 'statistics': { 'min': float, 'max': float, 'sum': float, 'mean': float, 'median': float, 'mode': [list[float], list[float], list[float]], 'median_absolute_deviation': float, 'variance': float, 'stddev': float, 't-test': { 't-statistic': float, 'conservative': {'deg_of_free': int, 'p-value': float}, 'welch': {'deg_of_free': float, 'p-value': float}}, 'psi': float, "chi2-test": { "chi2-statistic": float, "deg_of_free": int, "p-value": float }, 'unique_count': int, 'unique_ratio': float, 'categories': [list[str], list[str], list[str]], 'gini_impurity': float, 'unalikeability': float, 'categorical_count': [dict[str, int], dict[str, int], dict[str, int]], 'avg_predictions': [dict[str, float]], 'label_representation': [dict[str, float]], 'sample_size': int, 'null_count': int, 'null_types': [list[str], list[str], list[str]], 'null_types_index': [dict[str, int], dict[str, int], dict[str, int]], 'data_type_representation': [dict[str, float]] }, "null_replication_metrics": { "class_prior": list[int], "class_sum": list[list[int]], "class_mean": list[list[int]] } }] } ``` -------------------------------- ### Example: Plot Column Missing Values Barchart Source: https://capitalone.github.io/DataProfiler/graphs.html This example demonstrates plotting a barchart of a column's missing values. The function accepts a list of profilers. ```python import pandas as pd from dataprofiler.profilers.profile_builder import StructuredColProfiler from dataprofiler.reports import graphs data = pd.Series([1, 2, 3, None, None, 4], name='example', dtype=str) profiler = StructuredColProfiler(data) # Plot the axes, can be a list of multiple columns ax = graphs.plot_col_missing_values([profiler]) ``` -------------------------------- ### Install Slimmer DataProfiler Package Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Install a slimmer version of DataProfiler from PyPI if ML requirements are too strict or not needed. This version disables default sensitive data detection. ```console pip install DataProfiler ``` -------------------------------- ### Initialize ProfilerOptions with 'lower_memory_sketching' preset Source: https://capitalone.github.io/DataProfiler/profiler.html Initializes ProfilerOptions using the 'lower_memory_sketching' preset for memory-efficient profiling. ```python options = ProfilerOptions(presets="lower_memory_sketching") ``` -------------------------------- ### BaseProfilerOptions Source: https://capitalone.github.io/DataProfiler/dataprofiler.profilers.profiler_options.html Provides methods to get, set, and validate profiler options. ```APIDOC ## properties ### Description Return a copy of the option properties. ### Returns dictionary of the option’s properties attr: value ### Return type dict ## set ### Description Set all the options. Send in a dict that contains all of or a subset of the appropriate options. Set the values of the options. Will raise error if the formatting is improper. ### Parameters - **options** (_dict_) – dict containing the options you want to set. ### Returns None ### Return type None ## validate ### Description Validate the options do not conflict and cause errors. Raises error/warning if so. ### Parameters - **raise_error** (_bool_) – Flag that raises errors if true. Returns errors if false. ### Returns list of errors (if raise_error is false) ### Return type list(str) ``` -------------------------------- ### Get RegexModel Labels Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Retrieve a list of all labels currently managed by the RegexModel. This property is read-only. ```python labels_list = regex_model._labels ``` -------------------------------- ### CharacterLevelCnnModel.reset_weights Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.character_level_cnn_model.html Resets all trainable weights of the model to their initial values, useful for starting training from scratch. ```APIDOC ## CharacterLevelCnnModel.reset_weights() ### Description Resets all trainable weights of the model to their initial values. ### Method `reset_weights()` ``` -------------------------------- ### Get RegexModel Class Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Retrieve a specific subclass of BaseModel by its class name. This is an internal utility method. ```python model_class = RegexModel._get_class('SomeModelClassName') ``` -------------------------------- ### Initialize ParquetData with Options Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.parquet_data.html Initialize the ParquetData class with data or a file path and specify options for loading. Options include data format, selected columns, and header. ```python options = dict( data_format= type: str, choices: "dataframe", "records", "json" selected_columns= type: list(str) header= type: any ) parquet_data = ParquetData(data="path/to/your/data.parquet", options=options) ``` -------------------------------- ### dataprofiler.data_readers.data_utils.find_nth_loc Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.data_utils.html Searches for the nth occurrence of a substring within a given string and returns its start and end indices. ```APIDOC ## find_nth_loc ### Description Search string via search_query and return nth index in which query occurs. If there are less than ‘n’ the last loc is returned. ### Parameters * **string** (str | None) – Input string, to be searched. * **search_query** (str | None) – char(s) to find nth occurrence of. * **n** (int) – The number of occurrences to iterate through, defaults to 0. * **ignore_consecutive** (bool) – Ignore consecutive matches in the search query, defaults to True. ### Returns A tuple containing the index of the nth or last occurrence of the search_query and the count of occurrences. ``` -------------------------------- ### BaseOption.set Source: https://capitalone.github.io/DataProfiler/dataprofiler.profilers.profiler_options.html Sets all the options from a dictionary. Raises an error if the formatting is improper. ```APIDOC ## BaseOption.set ### Description Set all the options. Send in a dict that contains all of or a subset of the appropriate options. Set the values of the options. Will raise error if the formatting is improper. ### Parameters * **options** (dict) – dict containing the options you want to set. ### Returns None ``` -------------------------------- ### Import Graphs Module Source: https://capitalone.github.io/DataProfiler/_sources/graphs.rst.txt Import the necessary graphs module from the dataprofiler library. Ensure `dataprofiler[reports]` is installed. ```python from dataprofiler.reports import graphs ``` -------------------------------- ### Instantiate ParquetData class Source: https://capitalone.github.io/DataProfiler/data_readers.html Shows how to instantiate the ParquetData class for loading Parquet datasets, with options for data format, key selection, and row sampling. ```APIDOC ## Instantiate ParquetData class ### Description Directly instantiate the `ParquetData` class for loading Parquet datasets. Options include data format, selecting keys, and sampling rows. ### Method ```python from dataprofiler.data_readers.parquet_data import ParquetData data = ParquetData("your_file.parquet", options={"data_format": "dataframe"}) ``` ### Options for ParquetData - `data_format` (string): Format of the data, choices: "dataframe", "records", "json". - `selected_keys` (list): List of keys to select from the Parquet data. - `sample_nrows` (int): Number of rows to sample. ``` -------------------------------- ### BaseOption Class Source: https://capitalone.github.io/DataProfiler/dataprofiler.profilers.profiler_options.html The base class for configuring profiler options. It provides methods to get, set, and validate options. ```APIDOC ## BaseOption Class ### Description The base class for configuring profiler options. It provides methods to get, set, and validate options. ### Methods #### `_properties()` Returns a copy of the option properties as a dictionary. #### `set(options: dict[str, bool]) -> None` Sets all the options using a dictionary. Raises an error if the formatting is improper. #### `validate(raise_error: bool = True) -> list[str] | None` Validates that the options do not conflict. Raises errors if `raise_error` is true, otherwise returns a list of errors. #### `_load_from_dict(data: dict[str, Any], config: dict | None = None) -> BaseOption` Class method to parse attributes from a JSON dictionary into a BaseOption object. ``` -------------------------------- ### Instantiate AVROData class Source: https://capitalone.github.io/DataProfiler/data_readers.html Demonstrates how to instantiate the AVROData class for loading AVRO datasets, with options for data format and key selection. ```APIDOC ## Instantiate AVROData class ### Description Directly instantiate the `AVROData` class for loading AVRO datasets. Options include data format and selecting specific keys. ### Method ```python from dataprofiler.data_readers.avro_data import AVROData data = AVROData("your_file.avro", options={"data_format": "records"}) ``` ### Options for AVROData - `data_format` (string): Format of the data, choices: "dataframe", "records", "avro", "json", "flattened_dataframe". - `selected_keys` (list): List of keys to select from the AVRO data. ``` -------------------------------- ### TextData Initialization Options Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.text_data.html Options for initializing TextData, including data format and samples per line. 'data_format' must be 'text', and 'samples_per_line' controls chunking. ```python options = dict( data_format= type: str, choices: "text" samples_per_line= type: int ) ``` -------------------------------- ### Get RegexModel Label Mapping Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Retrieve the current mapping of labels to their encoded integer values. This property is read-only. ```python label_map = regex_model._label_mapping ``` -------------------------------- ### Get RegexModel Parameters Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Retrieve a dictionary of specified parameters from the RegexModel. If no parameters are listed, all available parameters are returned. ```python params = regex_model.get_parameters(param_list=['max_length', 'dim_embed']) ``` -------------------------------- ### Get Batch Generator for AVRO Data Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.avro_data.html Retrieves a generator for iterating over the AVRO data in batches of a specified size. ```python get_batch_generator(_batch_size : int_) → Generator[DataFrame | List, None, None] ``` -------------------------------- ### Download Popmon Tutorial Data Source: https://capitalone.github.io/DataProfiler/popmon_dp_loader_example.html Downloads the 'flight_delays.csv.gz' dataset from popmon's resources and saves it as 'flight_delays.csv'. ```python import gzip import shutil popmon_tutorial_data = popmon.resources.data("flight_delays.csv.gz") with gzip.open(popmon_tutorial_data, 'rb') as f_in: with open('./flight_delays.csv', 'wb') as f_out: shutil.copyfileobj(f_in, f_out) ``` -------------------------------- ### CharLoadTFModel.help() Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.char_load_tf_model.html Provides help information for the CharLoadTFModel. ```APIDOC ## CharLoadTFModel.help() ### Description Provides help information for the CharLoadTFModel. ### Method `help()` ### Parameters None ### Response Displays help documentation or returns a string with help information. ``` -------------------------------- ### Get RegexModel Number of Labels Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Retrieve the total number of unique labels currently configured in the RegexModel. This property is read-only. ```python num_labels = regex_model._num_labels ``` -------------------------------- ### BaseOption Methods Source: https://capitalone.github.io/DataProfiler/dataprofiler.profilers.profiler_options.html Provides methods for loading options from a dictionary, accessing properties, setting options, and validating configurations. ```APIDOC ## _classmethod _load_from_dict(_data_ , _config : dict | None = None_) ### Description Parse attribute from json dictionary into self. ### Parameters * **data** (_dict_ _[__string_ _,__Any_ _]_) – dictionary with attributes and values. * **config** (_Dict_ _|__None_) – config to override loading options params from dictionary ### Returns Options with attributes populated. ### Return type BaseOption ## _property _properties_ ### Description Return a copy of the option properties. ### Returns dictionary of the option’s properties attr: value ### Return type dict ## set(_options : dict[str, bool]_) ### Description Set all the options. Send in a dict that contains all of or a subset of the appropriate options. Set the values of the options. Will raise error if the formatting is improper. ### Parameters * **options** (_dict_) – dict containing the options you want to set. ### Returns None ## validate(_raise_error : bool = True_) ### Description Validate the options do not conflict and cause errors. Raises error/warning if so. ### Parameters * **raise_error** (_bool_) – Flag that raises errors if true. Returns errors if false. ### Returns list of errors (if raise_error is false) ### Return type list(str) ``` -------------------------------- ### Profile a Pandas DataFrame Source: https://capitalone.github.io/DataProfiler/overview.html Run the Data Profiler on a Pandas DataFrame and get a compact report. Ensure 'pandas' and 'dataprofiler' are imported. ```python import pandas as pd import dataprofiler as dp import json my_dataframe = pd.DataFrame([[1, 2.0],[1, 2.2],[-1, 3]], columns=["col_int", "col_float"]) profile = dp.Profiler(my_dataframe) report = profile.report(report_options={"output_format":"compact"}) # Print the report print(json.dumps(report, indent=4)) ``` -------------------------------- ### HyperLogLogOptions.set Source: https://capitalone.github.io/DataProfiler/dataprofiler.profilers.profiler_options.html Sets HyperLogLog options. ```APIDOC ## HyperLogLogOptions.set() ### Description Sets HyperLogLog options. ### Method `set(value: bool)` ### Parameters * **value** (bool) - Required - The boolean value to set. ``` -------------------------------- ### Save and Load Profile using JSON Source: https://capitalone.github.io/DataProfiler/_sources/profiler.rst.txt Illustrates saving a profile to a human-readable .json file and loading it back. This method currently only supports structured profiles. ```python import json from dataprofiler import Data, Profiler ``` -------------------------------- ### Import Libraries for Data Profiler Source: https://capitalone.github.io/DataProfiler/profiler_example.html Imports necessary libraries for using the DataProfiler. Ensure the dataprofiler library is installed and accessible in your Python path. ```python import os import sys import json try: sys.path.insert(0, '..') import dataprofiler as dp except ImportError: import dataprofiler as dp data_path = "../dataprofiler/tests/data" # remove extra tf loggin import tensorflow as tf tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR) ``` -------------------------------- ### Import Libraries for Demo Source: https://capitalone.github.io/DataProfiler/popmon_dp_loader_example.html Imports necessary libraries including dataprofiler, pandas, and popmon. Ensures dataprofiler can be imported from the parent directory if needed. ```python import os import sys try: sys.path.insert(0, '..') import dataprofiler as dp except ImportError: import dataprofiler as dp import pandas as pd import popmon # noqa ``` -------------------------------- ### Save and Load Profile using Pickle Source: https://capitalone.github.io/DataProfiler/_sources/profiler.rst.txt Demonstrates how to save a structured profile to a .pkl file and load it back into a profile object. This method is suitable for structured profiles. ```python import json from dataprofiler import Data, Profiler # Load a CSV file, with "," as the delimiter data = Data("your_file.csv") # Read data into profile profile = Profiler(data) # save structured profile to pkl file profile.save(filepath="my_profile.pkl") # load pkl file to structured profile loaded_pkl_profile = dp.Profiler.load(filepath="my_profile.pkl") print(json.dumps(loaded_pkl_profile.report(report_options={"output_format": "compact"}), indent=4)) ``` -------------------------------- ### Example: Plot Specific Column Histogram from StructuredProfiler Source: https://capitalone.github.io/DataProfiler/_sources/graphs.rst.txt Demonstrates plotting a histogram for a specific column (by name/index) from a StructuredProfiler. Requires `dataprofiler` and `dataprofiler.reports`. ```python import dataprofiler as dp from dataprofiler.reports import graphs data = [[1, 'a', 1.0], [2, 'b', 2.2], [3, 'c', 3.5], [None, 'd', 10.0]] profiler = dp.StructuredProfiler(data) # This will only plot the specified column, 0. columns_names = [0] fig = graphs.plot_histograms(profiler, columns_names) fig.show() ``` -------------------------------- ### Train Structured Labeler from Scratch Source: https://capitalone.github.io/DataProfiler/labeler.html Loads a small CSV dataset, splits it into training and testing sets, and trains a new structured labeler using column names as labels. The trained labeler is saved to a specified directory. ```python data = dp.Data("../dataprofiler/tests/data/csv/SchoolDataSmall.csv") df = data.data[["OPEID6", "INSTURL", "SEARCH_STRING"]] df.head() # split data to training and test set split_ratio = 0.2 df = df.sample(frac=1).reset_index(drop=True) data_train = df[:int((1 - split_ratio) * len(df))] data_test = df[int((1 - split_ratio) * len(df)):] # train a new labeler with column names as labels if not os.path.exists('data_labeler_saved'): os.makedirs('data_labeler_saved') labeler = dp.train_structured_labeler( data=data_train, save_dirpath="data_labeler_saved", epochs=10, default_label="OPEID6" ) ``` -------------------------------- ### Run Specific Unit Test File Source: https://capitalone.github.io/DataProfiler/_sources/install.rst.txt Execute unit tests from a specific file, such as 'test_profile_builder.py', using the 'unittest' discover command. ```console DATAPROFILER_SEED=0 python3 -m unittest discover -p test_profile_builder.py ``` -------------------------------- ### load Source: https://capitalone.github.io/DataProfiler/dataprofiler.profilers.profile_builder.html Class method to load a profiler from disk. ```APIDOC ## load ### Description Load profiler from disk. ### Parameters #### Query Parameters - **filepath** (str) - Path of file to load from - **load_method** (Optional[str]) - The desired loading method, default = None ### Returns BaseProfiler: Profiler being loaded, StructuredProfiler or UnstructuredProfiler ``` -------------------------------- ### Initialize ProfilerOptions with 'data_types' preset Source: https://capitalone.github.io/DataProfiler/profiler.html Initializes ProfilerOptions using the 'data_types' preset, focusing on data type analysis. ```python options = ProfilerOptions(presets="data_types") ``` -------------------------------- ### Plot Column Missing Values Barchart Source: https://capitalone.github.io/DataProfiler/_sources/graphs.rst.txt Use this snippet to plot a barchart of a column's missing values. Ensure pandas and dataprofiler are installed. ```python import pandas as pd from dataprofiler.profilers.profile_builder import StructuredColProfiler from dataprofiler.reports import graphs data = pd.Series([1, 2, 3, None, None, 4], name='example', dtype=str) profiler = StructuredColProfiler(data) # Plot the axes, can be a list of multiple columns ax = graphs.plot_col_missing_values([profiler]) # get and show the figure of the plotted histogram fig = ax.get_figure() fig.show() ``` -------------------------------- ### Import Libraries for Data Profiler Source: https://capitalone.github.io/DataProfiler/column_name_labeler_example.html Imports necessary libraries including pandas and dataprofiler. Includes a fallback for importing dataprofiler if it's not installed directly. ```python import os import sys import json from pprint import pprint import pandas as pd try: import dataprofiler as dp except ImportError: sys.path.insert(0, '../..') import dataprofiler as dp ``` -------------------------------- ### Save and Load PKL Profile Source: https://capitalone.github.io/DataProfiler/profiler.html Demonstrates saving a profile to a PKL file and then loading it back. Use this for efficient storage and retrieval of profile objects. ```python import json from dataprofiler import Data, Profiler # Load a CSV file, with "," as the delimiter data = Data("your_file.csv") # Read data into profile profile = Profiler(data) # save structured profile to pkl file profile.save(filepath="my_profile.pkl") # load pkl file to structured profile loaded_pkl_profile = dp.Profiler.load(filepath="my_profile.pkl") print(json.dumps(loaded_pkl_profile.report(report_options={"output_format": "compact"}), indent=4)) ``` -------------------------------- ### Explicitly Specify Structured Profiler Source: https://capitalone.github.io/DataProfiler/profiler_example.html Explicitly set the profiler type to 'structured' when initializing the Profiler. This example also demonstrates generating a 'pretty' report. ```python data = dp.Data(os.path.join(data_path, "csv/aws_honeypot_marx_geo.csv")) profile = dp.Profiler(data, profiler_type='structured') # print the report using json to prettify. report = profile.report(report_options={"output_format": "pretty"}) print(json.dumps(report, indent=4)) ``` -------------------------------- ### S3Helper.create_s3_client Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.data_utils.html Creates and returns an S3 client instance. ```APIDOC ## S3Helper.create_s3_client ### Description Create and return an S3 client. ### Method S3Helper.create_s3_client ### Parameters #### Path Parameters - **aws_access_key_id** (str | None) - Optional - The AWS access key ID. - **aws_secret_access_key** (str | None) - Optional - The AWS secret access key. - **aws_session_token** (str | None) - Optional - The AWS session token (optional, typically used for temporary credentials). - **region_name** (str | None) - Optional - The AWS region name (default is ‘us-east-1’). ### Returns A S3 client instance. ### Return Type boto3.client ``` -------------------------------- ### Import dataprofiler Source: https://capitalone.github.io/DataProfiler/data_reader.html This snippet shows how to import the dataprofiler library, handling potential import issues by adjusting the system path. ```python import os import sys try: sys.path.insert(0, '..') import dataprofiler as dp except ImportError: import dataprofiler as dp ``` -------------------------------- ### Instantiate JSONData class Source: https://capitalone.github.io/DataProfiler/data_readers.html Shows how to specifically instantiate the JSONData class for loading JSON datasets, with options for data format and key selection. ```APIDOC ## Instantiate JSONData class ### Description Directly instantiate the `JSONData` class for loading JSON datasets. Options include data format and selecting specific keys. ### Method ```python from dataprofiler.data_readers.json_data import JSONData data = JSONData("your_file.json", options={"data_format": "dataframe"}) ``` ### Options for JSONData - `data_format` (string): Format of the data, choices: "dataframe", "records", "json", "flattened_dataframe". - `selected_keys` (list): List of keys to select from the JSON data. - `payload_keys` (list): List of dictionary keys that represent the payload (defaults to ["data", "payload", "response"]). ``` -------------------------------- ### Data Profile Structure Source: https://capitalone.github.io/DataProfiler/_sources/index.rst.txt This is an example of the structured format for a data profile generated by the DataProfiler library. It includes global statistics about the dataset and detailed statistics for each column. ```python "global_stats": { "samples_used": int, "column_count": int, "row_count": int, "row_has_null_ratio": float, "row_is_null_ratio": float, "unique_row_ratio": float, "duplicate_row_count": int, "file_type": string, "encoding": string, "correlation_matrix": list[list[int]], (*) "chi2_matrix": list[list[float]], "profile_schema": dict[string, list[int]] }, "data_stats": [ { "column_name": string, "data_type": string, "data_label": string, "categorical": bool, "order": string, "samples": list[str], "statistics": { "sample_size": int, "null_count": int, "null_types": list[string], "null_types_index": dict[string, list[int]], "data_type_representation": dict[string, list[string]], "min": [null, float], "max": [null, float], "sum": float, "mode": list[float], "median": float, "median_absolute_deviation": float, "mean": float, "variance": float, "stddev": float, "skewness": float, "kurtosis": float, "num_zeros": int, "num_negatives": int, "histogram": { "bin_counts": list[int], "bin_edges": list[float], }, "quantiles": { int: float }, "vocab": list[char], "avg_predictions": dict[string, float], "data_label_representation": dict[string, float], "categories": list[str], "unique_count": int, "unique_ratio": float, "categorical_count": dict[string, int], "gini_impurity": float, "unalikeability": float, "precision": { 'min': int, 'max': int, 'mean': float, 'var': float, 'std': float, 'sample_size': int, 'margin_of_error': float, 'confidence_level': float }, "times": dict[string, float], "format": string }, "null_replication_metrics": { "class_prior": list[int], "class_sum": list[list[int]], ``` -------------------------------- ### Get Batch Generator for Text Data Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.text_data.html Retrieve a generator to iterate over the text data in batches of a specified size. This is useful for processing large datasets efficiently. ```python get_batch_generator(_batch_size : int_) → Generator[DataFrame | List, None, None] ``` -------------------------------- ### Get Parquet Data Batch Generator Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.parquet_data.html Retrieve a generator to iterate over the Parquet data in batches of a specified size. This is useful for processing large datasets efficiently. ```python batch_generator = parquet_data.get_batch_generator(batch_size=1000) for batch in batch_generator: # Process each batch of data ``` -------------------------------- ### IntOptions Class Initialization Source: https://capitalone.github.io/DataProfiler/dataprofiler.profilers.profiler_options.html Initializes options specifically for Integer Columns, enabling configuration of various statistical properties. ```APIDOC ## IntOptions Class Initialization ### Description Initializes options specifically for Integer Columns, enabling configuration of various statistical properties. ### Class `dataprofiler.profilers.profiler_options.IntOptions` ### Variables * **is_enabled** (_bool_) – boolean option to enable/disable the column. * **min** (_BooleanOption_) – boolean option to enable/disable min * **max** (_BooleanOption_) – boolean option to enable/disable max * **mode** (_ModeOption_) – option to enable/disable mode and set return count * **median** (_BooleanOption_) – boolean option to enable/disable median * **sum** (_BooleanOption_) – boolean option to enable/disable sum * **variance** (_BooleanOption_) – boolean option to enable/disable variance * **skewness** (_BooleanOption_) – boolean option to enable/disable skewness * **kurtosis** (_BooleanOption_) – boolean option to enable/disable kurtosis * **histogram_and_quantiles** (_BooleanOption_) – boolean option to enable/disable histogram_and_quantiles ``` -------------------------------- ### Get Batch Generator for JSON Data Source: https://capitalone.github.io/DataProfiler/dataprofiler.data_readers.json_data.html Retrieves a generator that yields data in batches of a specified size. Useful for processing large JSON files iteratively. ```python get_batch_generator(batch_size: int) ``` -------------------------------- ### _help Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.character_level_cnn_model.html Provides a description of alterable parameters. ```APIDOC ## _help ### Description Help describe alterable parameters. ### Returns None ``` -------------------------------- ### UnstructuredDataLabeler.help Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.data_labelers.html Describe alterable parameters. Input data formats for preprocessors. Output data formats for postprocessors. ```APIDOC ## help ### Description Describe alterable parameters. Input data formats for preprocessors. Output data formats for postprocessors. ### Returns None ``` -------------------------------- ### Configure JSON Data Path and Report Output Source: https://capitalone.github.io/DataProfiler/popmon_dp_loader_example.html Specify the path to the JSON file, the time index, and the report output directory. This setup is for processing JSON data. ```python path = "../dataprofiler/tests/data/json/math.json" time_index = "data.9" report_output_dir = "./popmon_output/math" ``` -------------------------------- ### Instantiate Data class with a URL Source: https://capitalone.github.io/DataProfiler/data_readers.html Demonstrates loading data from a URL using the generic Data class, including the ability to specify SSL verification options. ```APIDOC ## Instantiate Data class with a URL ### Description Load data from a URL using the `dp.Data` class. SSL verification can be controlled via the `options` parameter. ### Method ```python import dataprofiler as dp data = dp.Data("https://you_website.com/your_file.file", options={"verify_ssl": "True"}) ``` ``` -------------------------------- ### Get Reversed RegexModel Label Mapping Source: https://capitalone.github.io/DataProfiler/dataprofiler.labelers.regex_model.html Retrieve a reversed mapping of encoded integer values to their corresponding labels. This is useful for converting predicted indices back to label names. ```python reversed_map = regex_model._reverse_label_mapping ``` -------------------------------- ### Identify Entities in Structured Data Source: https://capitalone.github.io/DataProfiler/_sources/data_labeling.rst.txt Use this snippet to make predictions and get labels for cells in structured data. Ensure you have loaded your data and initialized the DataLabeler with 'structured' type. ```python import dataprofiler as dp # load data and data labeler data = dp.Data("your_data.csv") data_labeler = dp.DataLabeler(labeler_type='structured') # make predictions and get labels per cell predictions = data_labeler.predict(data) ```