### Installing SDV from Source (Development Setup) - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md This snippet provides commands to clone the SDV repository, navigate into it, switch to the main branch, and install all development dependencies. This is necessary for integrating and testing the new RDT release with the SDV project. ```bash git clone https://github.com/sdv-dev/SDV cd SDV git checkout main make install-develop ``` -------------------------------- ### Installing RDT from Source (Development Setup) - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md This snippet provides commands to clone the RDT repository, navigate into it, switch to the main branch, and install all development dependencies. It's the initial step for setting up the RDT project for release or development work. ```bash git clone https://github.com/sdv-dev/RDT cd RDT git checkout main make install-develop ``` -------------------------------- ### Installing RDT from Source (Stable Branch) Source: https://github.com/sdv-dev/rdt/blob/main/INSTALL.md This sequence of commands allows installing RDT directly from its source code. It involves cloning the GitHub repository, checking out the stable branch, and then executing `make install`, which is suitable for Unix-based systems like GNU/Linux and macOS. ```bash git clone https://github.com/sdv-dev/RDT cd RDT git checkout stable make install ``` -------------------------------- ### Installing RDT with pip Source: https://github.com/sdv-dev/rdt/blob/main/INSTALL.md This command installs the RDT library using pip, the Python package installer. It is the recommended and easiest method, pulling the latest stable release directly from PyPi. ```bash pip install rdt ``` -------------------------------- ### Installing RDT from Local Wheel Distribution - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md This command installs the RDT library from its locally generated wheel distribution file using 'pip'. It's used to verify that the distributed package can be successfully installed and used in a new environment before public release. ```bash pip install /path/to/rdt/dist/.whl ``` -------------------------------- ### Installing RDT for Development Source: https://github.com/sdv-dev/rdt/blob/main/INSTALL.md These commands are used to set up RDT for development purposes, enabling modifications to the source code. It involves cloning the repository, switching to the `main` branch, creating a new development branch, and then running `make install-develp`. ```bash git clone git@github.com:sdv-dev/RDT cd RDT git checkout main git checkout -b make install-develp ``` -------------------------------- ### Installing RDT with pip Source: https://github.com/sdv-dev/rdt/blob/main/README.md This snippet demonstrates how to install the RDT library using the pip package manager. It is recommended to use a virtual environment to prevent conflicts with other software. ```bash pip install rdt ``` -------------------------------- ### Installing RDT Distribution for SDV Integration - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md This command installs the RDT library from its locally generated wheel distribution file into the environment where SDV is being tested. This step ensures that the SDV project uses the newly released RDT version for compatibility checks. ```bash pip install /path/to/rdt/dist/.whl ``` -------------------------------- ### Installing RDT with conda Source: https://github.com/sdv-dev/rdt/blob/main/INSTALL.md This command installs the RDT library using conda, a package and environment manager. It fetches the latest stable release from the sdv-dev and conda-forge channels on Anaconda. ```bash conda install -c sdv-dev -c conda-forge rdt ``` -------------------------------- ### Applying USPhoneNumberTransformer in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This example demonstrates the full lifecycle of the `USPhoneNumberTransformer`. It initializes the transformer, fits it to sample phone number data, transforms the data into its components, and then reverses the transformation to reconstruct the original phone numbers, showing both input and output dataframes. ```Python In [1]: transformer = USPhoneNumberTransformer() data = pd.DataFrame({ 'phone_numbers': ['1-202-555-0191', '1-202-555-0151', '1-202-867-5309'] }) transformer.fit(data, ['phone_numbers']) transformed = transformer.transform(data) In [2]: transformed Out [2]: phone_numbers.area_code phone_numbers.exchange phone_numbers.line phone_numbers.country_code 0 1 202 555 0191 1 1 202 555 0151 2 1 202 867 5309 In [3] reverse_transformed = transformer.reverse_transform(transformed) In [4] reverse_transformed Out [4] phone_numbers 0 1-202-555-0191 1 1-202-555-0151 2 1-202-867-5309 ``` -------------------------------- ### Creating a Test Directory for RDT Distribution - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md These commands create and navigate into a new directory named 'rdt-test', which is intended to be outside the main RDT project directory. This setup provides an isolated environment for testing the generated RDT distribution files. ```bash mkdir rdt-test cd rdt-test ``` -------------------------------- ### Complete USPhoneNumberTransformer Class in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet presents the full `USPhoneNumberTransformer` class, consolidating all previously defined methods: `__init__`, `_fit`, `get_output_sdtypes`, `get_next_transformers`, `_transform`, and a partial `_reverse_transform`. It serves as a comprehensive example of a custom RDT transformer for converting phone number strings to structured numeric data and back. ```Python class USPhoneNumberTransformer(BaseTransformer): INPUT_SDTYPE = 'phone_number' def __init__(self): self.has_country_code = None def _fit(self, columns_data): number = ''.join(columns_data.loc[0].split('-')) self.has_country_code = len(number) == 11 def get_output_sdtypes(self): output_sdtypes = { 'area_code': 'categorical', 'exchange': 'integer', 'line': 'integer' } if self.has_country_code: output_sdtypes['country_code'] = 'categorical' return self._get_output_to_property(output_sdtypes) def get_next_transformers(self): next_transformers = { 'country_code': 'FrequencyEncoder', 'area_code': 'FrequencyEncoder' } if self.has_country_code: next_transformers['country_code'] = 'FrequencyEncoder' return self._get_output_to_property(next_transformers) def _transform(self, data): return data.str.split('-', expand=True) def _reverse_transform(self, data): if self.has_country_code: country_code = data.iloc[:, 0].astype('str') area_code = data.iloc[:, 1].astype('str') exchange = data.iloc[:, 2].astype('str') line = data.iloc[:, 3].astype('str') return country_code + '-' + area_code + '-' + exchange + '-' + line ``` -------------------------------- ### Creating and Activating a Python Virtual Environment - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md These commands create a new Python virtual environment using 'virtualenv' with a specified Python version (e.g., Python 3.6) and then activate it. This ensures that the RDT distribution is installed and tested in an isolated environment, preventing conflicts with system-wide packages. ```bash virtualenv -p $(which python3.6) .venv source .venv/bin/activate ``` -------------------------------- ### Installing RDT with conda Source: https://github.com/sdv-dev/rdt/blob/main/README.md This snippet shows how to install the RDT library using the conda package manager from the conda-forge channel. Using a virtual environment is advised to avoid conflicts. ```bash conda install -c conda-forge rdt ``` -------------------------------- ### Running All RDT Tests and Linting with Tox - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md This command executes all configured tests and linting checks for the RDT project using 'tox'. It's crucial for ensuring code quality and functionality across different Python environments before a release. Requires various Python versions installed as per 'tox.ini'. ```bash make test-all ``` -------------------------------- ### Implementing BaseTransformer Default Methods - Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet outlines the default methods available in the `BaseTransformer` class, which can be overridden by subclasses. These methods include `get_output_sdtypes()` and `get_next_transformers()` for managing output metadata, and `fit()`, `transform()`, `reverse_transform()` for the core data transformation lifecycle. The `fit` method prepares the transformer, `transform` applies the transformation, and `reverse_transform` reverts it. ```Python class BaseTransformer: def get_output_sdtypes(self) -> list: # Returns names of columns created by transform pass def get_next_transformers(self) -> dict: # Returns mapping of output columns to next transformers pass def fit(self, data: 'pandas.DataFrame', columns: list): # Stores information needed for transform pass def transform(self, data: 'pandas.DataFrame', drop: bool) -> 'pandas.DataFrame': # Transforms data, returns DataFrame pass def reverse_transform(self, data: 'pandas.DataFrame', drop: bool) -> 'pandas.DataFrame': # Reverses transformed data, returns DataFrame pass ``` -------------------------------- ### Running SDV Tests with New RDT Distribution - Bash Source: https://github.com/sdv-dev/rdt/blob/main/RELEASE.md This command executes the test suite for the SDV project. It's performed after installing the new RDT distribution to ensure that the updated RDT version is fully compatible and does not introduce regressions in SDV. ```bash make test ``` -------------------------------- ### Implementing Custom BaseTransformer Methods - Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet specifies the essential methods that must be implemented when creating a new `BaseTransformer` subclass. These include `_fit(columns_data)` for learning transformation parameters, `_transform(columns_data)` for applying the forward transformation, and `_reverse_transform()` for reverting the transformation. Attributes that cannot be determined until `fit` is called should be set within the `_fit` method. ```Python class CustomTransformer(BaseTransformer): def _fit(self, columns_data: 'pandas.DataFrame'): # Implementation for fitting the transformer pass def _transform(self, columns_data: 'pandas.DataFrame') -> 'pandas.DataFrame': # Implementation for transforming the data pass def _reverse_transform(self, transformed_data: 'pandas.DataFrame') -> 'pandas.DataFrame': # Implementation for reverse transforming the data pass ``` -------------------------------- ### Initializing USPhoneNumberTransformer in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet defines the `USPhoneNumberTransformer` class, inheriting from `BaseTransformer`. It sets the `INPUT_SDTYPE` to 'phone_number' and initializes the `has_country_code` attribute to `None` in the constructor, which will be determined during the fitting process. ```Python class USPhoneNumberTransformer(BaseTransformer): INPUT_SDTYPE = 'phone_number' def __init__(self): self.has_country_code = None ``` -------------------------------- ### Defining Performance Thresholds for USPhoneNumberGenerator in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet provides the implementation for the `get_performance_thresholds` static method within the `USPhoneNumberGenerator` class. It returns a dictionary specifying expected time (in seconds) and memory (in bytes) thresholds for the `fit` and `transform` methods, crucial for performance testing of the associated transformer. ```Python @staticmethod def get_performance_thresholds(): """Return the expected thresholds.""" return { 'fit': { 'time': 1, 'memory': 100.0 }, 'transform': { 'time': 1, 'memory': 1000.0 }, ``` -------------------------------- ### Using USPhoneNumberTransformer with HyperTransformer in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This code illustrates how to integrate `USPhoneNumberTransformer` within a `HyperTransformer` pipeline. It configures `HyperTransformer` to use the custom phone number transformer for a specific field, then demonstrates fitting, transforming, and reverse-transforming data, highlighting the end-to-end process. ```Python In [1]: ht = HyperTransformer( default_sdtype_transformers={'phone_number': USPhoneNumberTransformer}, field_sdtypes={'phone_numbers': 'phone_number'} ) ht.fit(data) transformed = ht.transform(data) In [2]: transformed Out [2]: phone_numbers.area_code.value phone_numbers.exchange phone_numbers.line phone_numbers.country_code.value 0 0.5 202 555 0.500000 1 0.5 202 555 0.166667 2 0.5 202 867 0.833333 In [3]: reverse_transformed = ht.reverse_transform(transformed) In [4]: reverse_transformed Out [4]: phone_numbers 0 1-202-555-0191 1 1-202-555-0151 2 1-202-867-5309 ``` -------------------------------- ### Fitting USPhoneNumberTransformer in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This `_fit` method determines if phone numbers include a country code. It inspects the first entry of `columns_data` by removing hyphens and checks if the resulting number string has a length of 11, setting `self.has_country_code` accordingly. ```Python def _fit(self, columns_data): number = ''.join(columns_data.loc[0].split('-')) self.has_country_code = len(number) == 11 ``` -------------------------------- ### Reconstructing US Phone Numbers in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet demonstrates how to reconstruct a US phone number string from its component parts (area code, exchange, line) by concatenating them with hyphens. It assumes the components are extracted from a pandas DataFrame and are of string type. ```Python area_code = data.iloc[:, 0].astype('str') exchange = data.iloc[:, 1].astype('str') line = data.iloc[:, 2].astype('str') return area_code + '-' + exchange + '-' + line ``` -------------------------------- ### Managing Table Transformations with HyperTransformer - Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet details the primary methods of the `HyperTransformer` class, designed for transforming entire tables. The `fit()` method identifies and fits appropriate transformers for each column, potentially recursively. The `transform()` method applies the sequence of fitted transformers, and `reverse_transform()` reverts the entire table transformation by calling transformers in reverse order. ```Python class HyperTransformer: def fit(self, data: 'pandas.DataFrame'): # Finds and fits transformers for each column/group pass def transform(self, data: 'pandas.DataFrame') -> 'pandas.DataFrame': # Applies sequence of fitted transformers pass def reverse_transform(self, data: 'pandas.DataFrame') -> 'pandas.DataFrame': # Reverses sequence of transformations pass ``` -------------------------------- ### Implementing USPhoneNumberGenerator's generate method in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet defines the `USPhoneNumberGenerator` class, inheriting from `BaseDatasetGenerator` and `ABC`. It sets the `SDTYPE` to 'phone_number' and implements the `generate` static method, which creates a NumPy array of random, synthetically formatted US phone numbers for a given number of rows. ```Python from abc import ABC import numpy as np from tests.datasets.base import BaseDatasetGenerator class USPhoneNumberGenerator(BaseDatasetGenerator, ABC): SDTYPE = 'phone_number' @staticmethod def generate(num_rows): area_codes = np.random.randint(low=100, high=999, size=num_rows).astype(str) exchange = np.random.randint(low=100, high=999, size=num_rows).astype(str) line = np.random.randint(low=1000, high=9999, size=num_rows).astype(str) return np.apply_along_axis('-'.join, 0, [area_codes, exchange, line]) ``` -------------------------------- ### Defining Output SDTypes and Next Transformers in Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst These methods dynamically define the output sdtypes and subsequent transformers based on the presence of a country code. `get_output_sdtypes` specifies the data types for 'area_code', 'exchange', and 'line', optionally adding 'country_code'. `get_next_transformers` assigns 'FrequencyEncoder' to 'country_code' and 'area_code', adapting to whether a country code is present. Both methods use `_get_output_to_property` to prepend column prefixes. ```Python def get_output_sdtypes(self): output_sdtypes = { 'area_code': 'categorical', 'exchange': 'integer', 'line': 'integer' } if self.has_country_code: output_sdtypes['country_code'] = 'categorical' return self._get_output_to_property(output_sdtypes) def get_next_transformers(self): next_transformers = { 'country_code': 'FrequencyEncoder', 'area_code': 'FrequencyEncoder' } if self.has_country_code: next_transformers['country_code'] = 'FrequencyEncoder' return self._get_output_to_property(next_transformers) ``` -------------------------------- ### Defining BaseTransformer Attributes - Python Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet describes the core attributes of the `BaseTransformer` class, which are essential for defining its behavior and state. These attributes include `INPUT_SDTYPE` for input data type, `output_properties` for transformed column metadata, `columns` and `output_columns` for tracking column names, and `column_prefix` for ensuring unique output column names. Some attributes are set during the `fit` method. ```Python class BaseTransformer: INPUT_SDTYPE: str output_properties: dict columns: list output_columns: list column_prefix: str ``` -------------------------------- ### Generating US Phone Numbers with USPhoneNumberGenerator (Python) Source: https://github.com/sdv-dev/rdt/blob/main/DEVELOPMENT.rst This snippet demonstrates how to use the `USPhoneNumberGenerator` to generate a specified number of synthetic US phone numbers. It calls the `generate` method with an integer argument (e.g., 100) to produce an array of unique phone number strings. The output is a NumPy array containing formatted phone number strings. ```Python In [1]: USPhoneNumberGenerator.generate(100) Out [1]: array(['160-919-7653', '347-212-8425', '717-820-4356', '483-675-6853', '656-141-2176', '681-981-5310', '314-989-4289', '138-343-6582', '406-683-8597', '639-156-5496', '625-600-1649', '110-477-8992', '770-731-6200', '166-491-9881', '418-682-9540', '889-169-1878', '660-213-4713', '270-506-9422', '323-691-2507', '189-158-5409', '605-218-6776', '944-980-8854', '773-290-6675', '969-724-8712', '617-979-3609', '145-828-6455', '570-923-8982', '260-800-5404', '301-453-3972', '454-629-5258', '298-394-6958', '700-285-1703', '439-683-2711', '935-387-1178', '151-643-7354', '549-741-6070', '617-142-6518', '759-653-4626', '482-778-1256', '909-538-2919', '772-617-8616', '691-559-2419', '274-200-5514', '744-163-6255', '760-709-7880', '909-782-6044', '826-607-6956', '902-609-2589', '345-796-8422', '818-867-9468', '430-906-3757', '143-788-5794', '340-705-3813', '211-447-7218', '912-799-7431', '840-211-5830', '752-600-1938', '236-659-2646', '591-946-1546', '903-564-4356', '928-847-8630', '315-775-9896', '384-323-8186', '192-282-8873', '861-497-3333', '839-304-2029', '674-261-5948', '721-642-9755', '761-787-2193', '429-720-9832', '126-876-2681', '327-533-3443', '170-210-5689', '916-945-8487', '619-332-6223', '515-453-5862', '509-666-4074', '231-687-8172', '489-862-2525', '602-456-5236', '549-936-9406', '471-989-5828', '424-436-1012', '405-996-8833', '786-811-5453', '851-897-7043', '462-381-9671', '328-267-1474', '482-171-7564', '245-353-7712', '589-535-6689', '864-252-5314', '990-737-8649', '112-189-9047', '126-316-8627', '985-724-3452', '119-612-8449', '456-529-1190', '344-956-1910', '125-962-2067'], dtype='