### Install GdprDump via Composer Source: https://github.com/smile-sa/gdpr-dump/wiki/Home Install the project using the Composer package manager. ```bash composer create-project --no-dev --prefer-dist smile/gdpr-dump ``` -------------------------------- ### Configure Dump Settings Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Example of customizing the dump section in the configuration file. ```yaml dump: hex_blob: true routines: true ``` -------------------------------- ### Install GdprDump via Phar Source: https://github.com/smile-sa/gdpr-dump/wiki/Home Download and prepare the GdprDump phar executable. ```bash wget https://github.com/Smile-SA/gdpr-dump/releases/latest/download/gdpr-dump.phar chmod +x gdpr-dump.phar ./gdpr-dump.phar --version ``` -------------------------------- ### Configure GdprDump Source: https://github.com/smile-sa/gdpr-dump/wiki/Home Example YAML configuration for database connection, table truncation, filtering, and data anonymization. ```yaml database: # Allows using environment variables to provide database credentials host: '%env(DB_HOST)%' user: '%env(DB_USER)%' password: '%env(DB_PASSWORD)%' name: '%env(DB_NAME)%' tables: '*_log': # Dump only the schema (no data) for all tables suffixed with "_log" truncate: true products: # Include only recently created products where: 'created_at > date_sub(now(), interval 60 day)' users: # Anonymize the data of the "users" table converters: email: converter: 'randomizeEmail' unique: true password: converter: 'randomizeText' # Skip anonymization for users registered with acme.com email domain skip_conversion_if: 'strpos({{email}}, "@acme.com") !== false' ``` -------------------------------- ### Configure Boolean Environment Variable for Table Truncation Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Set boolean environment variables using the `%env(bool:VARIABLE_NAME)%` syntax. This example configures table truncation. ```yaml tables: cache: truncate: '%env(bool:TRUNCATE_CACHE_TABLE)%' ``` -------------------------------- ### Randomize Email Converter Usage Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Randomizes email usernames and replaces domains with safe example domains. Supports custom domains and minimum length constraints. ```yaml tables: users: converters: email: converter: 'randomizeEmail' unique: true work_email: converter: 'randomizeEmail' parameters: domains: ['test.local', 'dev.example.com'] min_length: 5 ``` -------------------------------- ### Database Configuration with URL Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure database connection using a single database URL string. ```yaml database: url: 'mysql://user:password@localhost/db_name' ``` -------------------------------- ### Dump Settings: Triggers and Init Commands Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure dump settings to skip triggers and run initialization commands after connection. ```yaml dump: # Skip triggers skip_triggers: false # Run initialization commands after connection init_commands: - 'SET NAMES utf8mb4' - 'SET SESSION sql_mode = ""' ``` -------------------------------- ### Use Multiple Configuration Templates Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Combine multiple templates by listing their paths under the `extends` parameter. Configurations are merged in the order provided. ```yaml extends: - 'path/to/config1.yaml' - 'path/to/config2.yaml' ``` -------------------------------- ### Run tests with Docker Source: https://github.com/smile-sa/gdpr-dump/blob/5.x/CONTRIBUTING.md Executes the project analysis and test suite using the recommended Docker environment. ```bash make analyse test ``` -------------------------------- ### Configure database settings with environment variables Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Use environment variable placeholders within the YAML configuration to securely inject database credentials. ```yaml database: host: '%env(DB_HOST)%' user: '%env(DB_USER)%' password: '%env(DB_PASSWORD)%' name: '%env(DB_NAME)%' ``` -------------------------------- ### GdprDump with Command-line Credentials Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Provide database connection details directly via command-line arguments. ```bash ./gdpr-dump.phar --host=localhost --user=root --database=mydb config.yaml > dump.sql ``` -------------------------------- ### Compile the Phar File Source: https://github.com/smile-sa/gdpr-dump/wiki/Phar-File-Compilation Use these commands to build the phar file with either all locales or a specific subset. ```bash make compile ``` ```bash make compile c="--locale=en_US" ``` -------------------------------- ### Inherit Multiple Templates Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Combine multiple configuration files or templates by providing a list to the extends key. ```yaml extends: - 'magento2' - 'path/to/custom-converters.yaml' - 'path/to/company-standards.yaml' version: '2.4.7' database: url: '%env(DATABASE_URL)%' ``` -------------------------------- ### Basic GdprDump Command Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Execute GdprDump with a configuration file to generate an anonymized database dump to standard output. ```bash ./gdpr-dump.phar config.yaml > dump.sql ``` -------------------------------- ### Database Configuration with PDO Driver Options Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure specific PDO driver options for the database connection. ```yaml database: host: 'localhost' user: 'root' password: 'secret' name: 'mydb' driver_options: 1002: 'SET NAMES utf8mb4' # PDO::MYSQL_ATTR_INIT_COMMAND ``` -------------------------------- ### Data Converter with Parameters Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Configure a data converter with specific parameters, such as domains for email randomization. ```yaml tables: my_table: converters: my_column: converter: 'randomizeEmail' parameters: domains: ['example.org'] ``` -------------------------------- ### Dump Settings: Routines and Blobs Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure dump settings to include stored procedures/functions and handle binary data correctly. ```yaml dump: # Include stored procedures and functions routines: true # Handle binary data properly hex_blob: true ``` -------------------------------- ### Compress Database Dumps Source: https://github.com/smile-sa/gdpr-dump/wiki/Home Pipe the dump command output into compression utilities. ```bash | gzip -9 > dump-$(date +%Y%m%d%H%M%S).gz ``` ```bash | bzip2 -c -9 > dump-$(date +%Y%m%d%H%M%S).bz2 ``` -------------------------------- ### Configure Environment Variables Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Inject environment variables using the %env() syntax, supporting type casting for booleans, integers, and JSON. ```yaml database: host: '%env(DB_HOST)%' user: '%env(DB_USER)%' password: '%env(DB_PASSWORD)%' name: '%env(DB_NAME)%' port: '%env(int:DB_PORT)%' # Type casting examples tables: cache: # Boolean type truncate: '%env(bool:TRUNCATE_CACHE)%' # JSON arrays from environment tables_blacklist: '%env(json:TABLES_BLACKLIST)%' # TABLES_BLACKLIST='["table1", "table2", "table3"]' # Available types: string (default), bool, int, float, json ``` -------------------------------- ### Generate Database Dump Source: https://github.com/smile-sa/gdpr-dump/wiki/Home Execute the phar file to generate a SQL dump file based on a configuration file. ```bash ./gdpr-dump.phar --host=db_host --user=db_user --database=db_name config.yaml > dump-$(date +%Y%m%d%H%M%S).sql ``` -------------------------------- ### Apply Version-Conditional Configuration Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use the 'if_version' key to apply specific table configurations based on the application version range. ```yaml extends: 'magento2' version: '2.4.7' # Base configuration applies to all versions tables: users: converters: email: converter: 'randomizeEmail' # Version-specific configuration if_version: '>=2.2': tables: # Tables that exist only in Magento 2.2+ email_abandoned_cart: converters: email: converter: 'randomizeEmail' '<2.2': tables: # Different data format in older versions quote_payment: converters: additional_information: converter: 'serializedData' ``` -------------------------------- ### Extend Custom Configuration Template Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Use the `extends` parameter with a path to a custom YAML configuration file. This merges the custom template with the current configuration. ```yaml extends: 'path/to/config.yaml' ``` -------------------------------- ### Table Whitelist Configuration Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Specify a list of tables to be included in the dump. ```yaml tables_whitelist: - 'users' - 'orders' - 'products' ``` -------------------------------- ### Configure Database Connection via Environment Variables Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Use the `%env(VARIABLE_NAME)%` syntax to inject environment variable values into database connection settings. Supports string, bool, int, float, and json types. ```yaml database: host: '%env(DB_HOST)%' user: '%env(DB_USER)%' password: '%env(DB_PASSWORD)%' name: '%env(DB_NAME)%' ``` -------------------------------- ### Configure Database Password via Environment Variable Source: https://github.com/smile-sa/gdpr-dump/wiki/Guidelines Use environment variables in the YAML configuration to avoid exposing passwords in command-line arguments. ```yaml database: password: '%env(DB_PASSWORD)%' ``` -------------------------------- ### Extend Predefined Templates Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Inherit from built-in templates like magento2 to simplify configuration for common frameworks. ```yaml # Magento 2 template (requires version) extends: 'magento2' version: '2.4.7' database: host: '%env(DB_HOST)%' user: '%env(DB_USER)%' password: '%env(DB_PASSWORD)%' name: '%env(DB_NAME)%' # Custom overrides tables: # Skip anonymization for internal admin accounts admin_user: skip_conversion_if: 'strpos({{email}}, "@mycompany.com") !== false' # Add filters for large tables quote: where: 'created_at > date_sub(now(), interval 60 day)' sales_order: where: 'created_at > date_sub(now(), interval 60 day)' ``` -------------------------------- ### Dump Settings: Insert and Locks Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure dump settings for extended inserts and foreign key lock handling. ```yaml dump: # Use extended INSERT statements for faster imports extended_insert: true # Disable foreign key checks during import add_locks: true ``` -------------------------------- ### Dump Settings: Transaction and Comments Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure dump settings for single transaction consistency and comment handling. ```yaml dump: # Use single transaction for consistent snapshot single_transaction: true # Skip comments in output skip_comments: false ``` -------------------------------- ### Table Data Filtering: Limit, WHERE, and Order By Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Combine row limiting, WHERE conditions, and ordering for complex data filtering. ```yaml tables: # Combine limit, where, and ordering products: where: 'status = "active"' order_by: 'created_at desc, id asc' limit: 5000 ``` -------------------------------- ### Compressed Dump with Gzip Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Pipe the output of GdprDump to gzip for compressed storage. ```bash ./gdpr-dump.phar config.yaml | gzip -9 > dump-$(date +%Y%m%d%H%M%S).sql.gz ``` -------------------------------- ### Use SQL Variables in Configuration Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Define SQL queries in the variables section to reference results in table filters and converter conditions. ```yaml # Define SQL variables variables: firstname_attr_id: 'SELECT attribute_id FROM eav_attribute WHERE attribute_code = "firstname" AND entity_type_id = 1' lastname_attr_id: 'SELECT attribute_id FROM eav_attribute WHERE attribute_code = "lastname" AND entity_type_id = 1' active_store_id: 'SELECT store_id FROM store WHERE code = "default"' tables: # Use variables in WHERE filters customer_entity_varchar: where: 'attribute_id IN (@firstname_attr_id, @lastname_attr_id)' # Use variables in converter conditions eav_attribute_value: converters: value: converter: 'anonymizeText' condition: '{{attribute_id}} == @firstname_attr_id || {{attribute_id}} == @lastname_attr_id' # Use in regular filters orders: where: 'store_id = @active_store_id' ``` -------------------------------- ### Compressed Dump with Bzip2 Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Pipe the output of GdprDump to bzip2 for compressed storage. ```bash ./gdpr-dump.phar config.yaml | bzip2 -c -9 > dump-$(date +%Y%m%d%H%M%S).sql.bz2 ``` -------------------------------- ### Configure Chain Converter Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Execute multiple converters in sequence on a single column, optionally using conditions to determine which converter to apply. ```yaml tables: users: converters: # Apply multiple transformations email: converter: 'chain' parameters: converters: - converter: 'randomizeEmail' - converter: 'toLower' # Conditional chain based on other columns value: converter: 'chain' parameters: converters: - converter: 'anonymizeText' condition: '{{type}} == 0' - converter: 'randomizeText' condition: '{{type}} == 1' - converter: 'setNull' condition: '{{type}} == 2' ``` -------------------------------- ### Prepend text converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Adds a specified prefix to every value in the column. ```yaml tables: my_table: converters: my_column: converter: 'prependText' parameters: value: 'test_' ``` -------------------------------- ### GdprDump with Progress Bar Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use the -v flag to display a progress bar during the dump process. Includes dynamic filename. ```bash ./gdpr-dump.phar -v --host=db_host --user=db_user --database=db_name config.yaml > dump-$(date +%Y%m%d%H%M%S).sql ``` -------------------------------- ### Define Table Whitelist Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Specify a list of tables to be included in the dump. ```yaml tables_whitelist: - 'table1' - 'table2' ``` -------------------------------- ### Replace Text Patterns Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use 'replace' for simple string substitutions and 'regexReplace' for pattern-based replacements using regular expressions. 'limit' can control the number of replacements. ```yaml tables: users: converters: # Simple string replacement comment: converter: 'replace' parameters: search: 'confidential' replacement: '[REDACTED]' # Regex replacement - replace all phone numbers notes: converter: 'regexReplace' parameters: pattern: '/\d{3}-\d{3}-\d{4}/' replacement: 'XXX-XXX-XXXX' # Regex with limited replacements description: converter: 'regexReplace' parameters: pattern: '/[0-9]+/' replacement: '***' limit: 2 ``` -------------------------------- ### Configure JSON Environment Variable for Blacklisted Tables Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Define array values using the `%env(json:VARIABLE_NAME)%` syntax. The environment variable should contain a JSON-formatted string representing the array. ```yaml tables_blacklist: '%env(json:TABLES_BLACKLIST)%' ``` -------------------------------- ### Define Tables for Dump Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Specify tables to be included in the dump. Wildcards can be used for table names. ```yaml tables: table1: # ... table2: # ... ``` -------------------------------- ### Generate Random Text and Emails Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Utilize 'randomText' for generating random strings with length constraints and 'randomEmail' for creating fake email addresses. Custom domains can be specified for email generation. ```yaml tables: users: converters: # Random text between 3-16 characters api_key: converter: 'randomText' # Random text with length constraints token: converter: 'randomText' parameters: min_length: 32 max_length: 64 # Random email generation temp_email: converter: 'randomEmail' parameters: domains: ['temp.example.com'] min_length: 8 max_length: 12 ``` -------------------------------- ### Configure Faker Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Use Faker formatters to generate fake data. The {{value}} placeholder can be used to pass the original column value as an argument. ```yaml tables: my_table: converters: my_column: converter: 'faker' parameters: formatter: 'numberBetween' arguments: [1, 100] ``` ```yaml tables: my_table: converters: my_column: converter: 'faker' parameters: formatter: 'shuffle' arguments: ['{{value}}'] ``` -------------------------------- ### Configure FromContext Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Retrieve values from the processing context, such as previously processed column data. ```yaml tables: my_table: converters: email: converter: 'randomizeEmail' email_lowercase: converter: 'chain' parameters: converters: - converter: 'fromContext' parameters: key: 'processed_data.email' - converter: 'toLower' ``` -------------------------------- ### Apply Hash Algorithms Source: https://context7.com/smile-sa/gdpr-dump/llms.txt The 'hash' converter applies various hashing algorithms like SHA1 (default), SHA256, or MD5 to column values. Useful for password or secret anonymization. ```yaml tables: users: converters: # SHA1 hash (default) password: converter: 'hash' # SHA256 hash api_secret: converter: 'hash' parameters: algorithm: 'sha256' # MD5 hash legacy_token: converter: 'hash' parameters: algorithm: 'md5' ``` -------------------------------- ### Run tests manually Source: https://github.com/smile-sa/gdpr-dump/blob/5.x/CONTRIBUTING.md Executes the full suite of manual quality assurance and testing tools including linting, style checking, static analysis, and unit tests. ```bash vendor/bin/parallel-lint src tests vendor/bin/phpcs vendor/bin/phpstan analyse vendor/bin/phpunit ``` -------------------------------- ### GdprDump Dry Run Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Perform a dry run to validate the configuration without creating a dump file. ```bash ./gdpr-dump.phar --dry-run config.yaml ``` -------------------------------- ### Table Blacklist Configuration Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Specify a list of tables to be excluded from the dump, supporting wildcards. ```yaml tables_blacklist: - 'sessions' - 'cache' - 'cache_*' # Wildcard: all tables starting with cache_ - '*_tmp' # Wildcard: all tables ending with _tmp - '*_log' # Wildcard: all log tables ``` -------------------------------- ### Configure Faker Converter Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use the faker converter to generate realistic fake data for specific columns. Supports locale configuration and passing arguments to formatters. ```yaml tables: users: converters: # Generate random first name firstname: converter: 'faker' parameters: formatter: 'firstName' # Generate random last name lastname: converter: 'faker' parameters: formatter: 'lastName' # Generate IPv4 address ip_address: converter: 'faker' parameters: formatter: 'ipv4' # Formatter with arguments age: converter: 'faker' parameters: formatter: 'numberBetween' arguments: [18, 80] # Using original value as argument shuffled_name: converter: 'faker' parameters: formatter: 'shuffle' arguments: ['{{value}}'] # Set faker locale for localized data faker: locale: 'de_DE' ``` -------------------------------- ### Prepend and Append Text Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Add prefixes with 'prependText' or suffixes with 'appendText' to existing string values. Useful for anonymization or categorization. ```yaml tables: users: converters: # Add prefix - "user1" becomes "test_user1" username: converter: 'prependText' parameters: value: 'test_' # Add suffix - "user1" becomes "user1_anon" email: converter: 'appendText' parameters: value: '_anonymized' ``` -------------------------------- ### Configure Default Faker Locale Source: https://github.com/smile-sa/gdpr-dump/wiki/Phar-File-Compilation Update the services.yaml file to change the default locale used by Faker. ```yaml parameters: # ... faker.locale: 'fr_FR' ``` -------------------------------- ### Generate Random Dates and DateTimes Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use 'randomDate' for dates and 'randomDateTime' for datetimes. Specify year ranges and formats as needed. 'null' for max_year defaults to the current year. ```yaml tables: users: converters: # Random date between 1900 and current year random_date: converter: 'randomDate' # Random date with year constraints registration_date: converter: 'randomDate' parameters: min_year: 2020 max_year: 2024 format: 'Y-m-d' # Random datetime with custom format last_login: converter: 'randomDateTime' parameters: min_year: 2023 max_year: null # null means current year format: 'Y-m-d H:i:s' ``` -------------------------------- ### Number Between Generator Configuration Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Configure the numberBetween converter to generate random integers within a specified range. Both minimum and maximum values are required parameters. ```yaml tables: my_table: converters: my_column: converter: 'numberBetween' parameters: min: 0 max: 100 ``` -------------------------------- ### Configure fromContext Converter Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Access previously converted values or original row data during the conversion process. ```yaml tables: users: converters: # First anonymize the email email: converter: 'randomizeEmail' # Then create lowercase version from processed email email_lowercase: converter: 'chain' parameters: converters: - converter: 'fromContext' parameters: key: 'processed_data.email' - converter: 'toLower' # Access original (unconverted) value original_domain: converter: 'fromContext' parameters: key: 'row_data.email' ``` -------------------------------- ### Order Rows in Table Dump Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Define the order in which rows are dumped from a table using SQL-like syntax in 'order_by'. ```yaml tables: my_table: order_by: 'sku, entity_id desc' ``` -------------------------------- ### Data Converters Reference Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters A collection of available converters for data transformation within the GDPR Dump configuration. ```APIDOC ## setValue ### Description Always returns the same specified value. ### Parameters - **value** (scalar/null) - Required - The value to set. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'setValue' parameters: value: 0 ``` ## toLower ### Description Converts all characters to lower case. ## toUpper ### Description Converts all characters to upper case. ## prependText ### Description Adds a prefix to every value. ### Parameters - **value** (string) - Required - The value to prepend. ## appendText ### Description Adds a suffix to every value. ### Parameters - **value** (string) - Required - The value to append. ## replace ### Description Replaces all occurrences of the search string with the replacement string. ### Parameters - **search** (string) - Required - The text to replace. - **replacement** (string) - Optional - The replacement text (default: ''). ## regexReplace ### Description Performs a regular expression search and replace. ### Parameters - **pattern** (string) - Required - The pattern to find. - **replacement** (string) - Optional - The replacement text (default: ''). - **limit** (int) - Optional - The max number of replacements (default: -1). ## hash ### Description Applies a hash algorithm on the value. ### Parameters - **algorithm** (string) - Required - The algorithm to use (e.g., md5, sha1, sha256). ``` -------------------------------- ### Configure Chain Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Execute a sequence of converters. Individual converters in a chain can be disabled by index in child configurations. ```yaml tables: my_table: converters: my_column: converter: 'chain' parameters: converters: - converter: 'anonymizeText' condition: '{{another_column}} == 0' - converter: 'randomizeText' condition: '{{another_column}} == 1' ``` ```yaml tables: my_table: converters: my_column: parameters: converters: 1: disabled: true ``` -------------------------------- ### Apply Conditional Converters in YAML Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use the condition key with PHP expressions to apply converters only when specific column criteria are met. ```yaml tables: users: converters: # Only convert if another column matches condition email: converter: 'randomizeEmail' condition: '{{is_customer}} === true' # Multiple conditions using PHP expressions phone: converter: 'randomizeNumber' condition: '{{country}} === "US" && {{phone}} !== null' # Conditional based on string content notes: converter: 'anonymizeText' condition: 'strpos({{notes}}, "confidential") !== false' ``` -------------------------------- ### Configure randomizeEmail Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters This converter randomizes the username part of an email and replaces the domain with a safe one. You can configure 'domains', 'min_length', and 'replacements'. ```yaml tables: my_table: converters: my_column: converter: 'randomizeEmail' ``` -------------------------------- ### Truncate Table Dump Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Configure a table to be dumped without any data by setting 'truncate' to true. ```yaml tables: my_table: truncate: true ``` -------------------------------- ### Configure Full Magento 2 Database Anonymization Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Define database credentials, dump settings, and table-specific anonymization rules including conditional skipping and row filtering. ```yaml extends: 'magento2' version: '2.4.7' database: host: '%env(DB_HOST)%' user: '%env(DB_USER)%' password: '%env(DB_PASSWORD)%' name: '%env(DB_NAME)%' dump: routines: true hex_blob: true # Skip anonymization for internal accounts tables: admin_user: skip_conversion_if: 'strpos({{email}}, "@mycompany.fr") !== false' # Dump only recent quotes for performance quote: where: 'created_at > date_sub(now(), interval 60 day)' # Dump only recent orders sales_order: where: 'created_at > date_sub(now(), interval 60 day)' # Anonymize custom extension table custom_reviews: converters: reviewer_email: converter: 'randomizeEmail' unique: true reviewer_name: converter: 'randomizeText' review_content: converter: 'anonymizeText' ``` -------------------------------- ### Data Converter with Condition Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Apply a data converter only when a specified PHP condition evaluates to true. Column values are available using double brackets. ```yaml tables: my_table: converters: my_column: converter: 'randomizeEmail' condition: '{{another_column}} !== null' ``` -------------------------------- ### Configure anonymizeEmail Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Anonymizes email addresses by applying anonymization to the username and replacing the domain. Configuration options include 'domains', 'replacement', 'delimiters', and 'min_word_length'. ```yaml tables: my_table: converters: my_column: converter: 'anonymizeEmail' ``` -------------------------------- ### Random Text Generator Configuration Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Configure the randomText converter to generate random text strings. Specify minimum and maximum lengths, and optionally provide a custom set of characters to use for generation. ```yaml tables: my_table: converters: my_column: converter: 'randomText' parameters: min_length: 0 max_length: 10 ``` -------------------------------- ### Generate Numbers Between a Range Source: https://context7.com/smile-sa/gdpr-dump/llms.txt The 'numberBetween' converter generates random integers within a specified minimum and maximum range. Useful for quantities or prices. ```yaml tables: products: converters: # Random integer between min and max stock_quantity: converter: 'numberBetween' parameters: min: 0 max: 1000 # Random price value price: converter: 'numberBetween' parameters: min: 100 max: 9999 ``` -------------------------------- ### Share Converter Results with cache_key Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use cache_key to maintain consistent anonymized values across different tables. ```yaml tables: # Primary table where value is first generated customer_entity: converters: email: converter: 'randomizeEmail' cache_key: 'customer_email' unique: true # Related tables use same anonymized email customer_grid_flat: converters: email: converter: 'randomizeEmail' cache_key: 'customer_email' unique: true newsletter_subscriber: converters: subscriber_email: converter: 'randomizeEmail' cache_key: 'customer_email' unique: true ``` -------------------------------- ### Convert string case converters Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Transforms column values to lower or upper case. ```yaml tables: my_table: converters: my_column: converter: 'toLower' ``` ```yaml tables: my_table: converters: my_column: converter: 'toUpper' ``` -------------------------------- ### Set Null Converter Configuration Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Use the setNull converter to set all values in a specified column to null. ```yaml tables: my_table: converters: my_column: converter: 'setNull' ``` -------------------------------- ### Chain Multiple Data Converters Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Use the 'chain' converter to apply multiple converters sequentially to a column, each with its own condition. ```yaml tables: my_table: converters: my_column: converter: 'chain' parameters: converters: - converter: 'anonymizeText' condition: '{{another_column}} == 0' - converter: 'randomizeText' condition: '{{another_column}} == 1' ``` -------------------------------- ### Filter Orders by Creation Date Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Filters orders created within the last 30 days. Related tables are automatically filtered via foreign key relationships. ```yaml tables: orders: where: 'created_at > date_sub(now(), interval 30 day)' ``` -------------------------------- ### Unset Table Configuration by Setting to Null Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Remove an entire table's configuration, including converters and filters, by setting the table key to `null`. This is only allowed if the value is defined in a parent config. ```yaml extends: 'magento2' tables: admin_user: ~ ``` -------------------------------- ### Anonymize Date Configuration Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Use this configuration to anonymize date values in a column. The day and month are randomized, while the year remains unchanged. Ensure the input date format matches the specified 'format' parameter. ```yaml tables: my_table: converters: my_column: converter: 'anonymizeDate' ``` -------------------------------- ### Define Table Blacklist Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Specify a list of tables to be excluded from the dump. ```yaml tables_blacklist: - 'table1' - 'table2' ``` -------------------------------- ### Limit Rows in Table Dump Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Set a maximum number of rows to dump for a table using the 'limit' property. Must be greater than 0. ```yaml tables: my_table: limit: 10000 ``` -------------------------------- ### Unset All Converters for a Table by Setting to Null Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Remove all converters for a specific table by setting the `converters` key to `null`. This requires the `converters` key to be defined in a parent configuration. ```yaml extends: 'magento2' tables: admin_user: converters: ~ ``` -------------------------------- ### Ensure Data Consistency Across Tables Source: https://github.com/smile-sa/gdpr-dump/wiki/Guidelines Use a cache_key to ensure anonymized values remain consistent across different database tables. ```yaml tables: customer_entity: converters: email: cache_key: 'customer_email' unique: true customer_flat_grid: converters: email: cache_key: 'customer_email' unique: true # ... repeat this for each table that stores a customer email ``` -------------------------------- ### Define SQL Variables for Dynamic Queries Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Store the results of SQL queries in user-defined variables. These variables can then be referenced in table filters and converter conditions. ```yaml variables: firstname_attribute_id: 'select attribute_id from eav_attribute where attribute_code = "firstname" and entity_type_id = 1' lastname_attribute_id: 'select attribute_id from eav_attribute where attribute_code = "lastname" and entity_type_id = 1' ``` -------------------------------- ### Anonymize Date and DateTime Converters Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Randomizes day and month while preserving the year for date anonymization. Supports custom date formats and anonymizes both date and time components. ```yaml tables: users: converters: date_of_birth: converter: 'anonymizeDate' birth_date: converter: 'anonymizeDate' parameters: format: 'd/m/Y' created_at: converter: 'anonymizeDateTime' ``` -------------------------------- ### Set Null Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Converts all values to null. ```APIDOC ## Set Null Converter ### Description Converts all values to `null`. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'setNull' ``` ``` -------------------------------- ### Set Fixed Values or NULL Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Use 'setNull' to set a column to NULL. 'setValue' can assign any fixed value, including numbers, strings, or booleans. ```yaml tables: users: converters: # Set to NULL reset_token: converter: 'setNull' # Set to fixed value login_attempts: converter: 'setValue' parameters: value: 0 # Set to fixed string status: converter: 'setValue' parameters: value: 'inactive' # Set to boolean is_verified: converter: 'setValue' parameters: value: false ``` -------------------------------- ### Configure randomizeText Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Use this converter to replace all characters in a column with random alphanumeric characters. Specify 'min_length' for the minimum length and 'replacements' for the character set. ```yaml tables: my_table: converters: my_column: converter: 'randomizeText' ``` -------------------------------- ### Append text converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Adds a specified suffix to every value in the column. ```yaml tables: my_table: converters: my_column: converter: 'appendText' parameters: value: '_test' ``` -------------------------------- ### Use SQL Variables in Converter Conditions Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Employ SQL variables in converter conditions using the `{{attribute_id}} == @variable_name` syntax. This allows for dynamic conditional anonymization. ```yaml tables: customer_entity_varchar: converters: value: converter: 'anonymizeText' condition: '{{attribute_id}} == @firstname_attribute_id' ``` -------------------------------- ### Hash value converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Applies a hashing algorithm to the column value. ```yaml tables: my_table: converters: my_column: converter: 'hash' parameters: algorithm: 'sha256' ``` -------------------------------- ### Configure randomizeNumber Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Use this converter to replace all numeric characters with random numbers, leaving other characters unchanged. It's configured by specifying the converter type. ```yaml tables: my_table: converters: my_column: converter: 'randomizeNumber' ``` -------------------------------- ### Randomize Text Converter Usage Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Replaces characters with random alphanumeric characters, preserving length. Can be configured with minimum length and custom replacement characters. ```yaml tables: users: converters: firstname: converter: 'randomizeText' username: converter: 'randomizeText' parameters: min_length: 5 unique: true password_hint: converter: 'randomizeText' parameters: replacements: 'abcdefghijklmnopqrstuvwxyz0123456789' ``` -------------------------------- ### Random Date Generator Configuration Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Configure the randomDate converter to generate random dates. You can specify the desired date format and the range of years (min_year, max_year) for the generated dates. ```yaml tables: my_table: converters: my_column: converter: 'randomDate' parameters: min_year: 2000 max_year: 2050 ``` -------------------------------- ### Randomizers Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Converters that replace parts of input values with random characters. ```APIDOC ## Randomize Text ### Description Converts all characters to random alphanumeric characters. ### Parameters #### Query Parameters - **min_length** (integer) - Optional - The minimum length of the generated value (when not empty). Defaults to `3`. - **replacements** (string) - Optional - A string that contains the replacement characters. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'randomizeText' ``` ``` ```APIDOC ## Randomize Email ### Description Applies transformations to anonymize email addresses by randomizing the username and using a safe domain. ### Parameters #### Query Parameters - **domains** (array of strings) - Optional - A list of email domains. Defaults to `['example.com', 'example.net', 'example.org']`. - **min_length** (integer) - Optional - The minimum length of the generated username (when not empty). Defaults to `3`. - **replacements** (string) - Optional - A string that contains the replacement characters. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'randomizeEmail' ``` ``` ```APIDOC ## Randomize Number ### Description Converts all numeric characters to random numbers, leaving other characters unmodified. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'randomizeNumber' ``` ``` -------------------------------- ### Table Data Filtering: Truncate Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure tables to be truncated, dumping only the schema without data. ```yaml tables: # Truncate table (dump schema only, no data) cache: truncate: true # Truncate all log tables using wildcard '*_log': truncate: true ``` -------------------------------- ### Regex replace converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Performs search and replace using regular expressions. ```yaml tables: my_table: converters: my_column: converter: 'regexReplace' parameters: pattern: '/[0-9]+/' replacement: '15' ``` -------------------------------- ### Transform Text Case Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Convert text to lowercase using 'toLower' or to uppercase using 'toUpper'. Applicable to string-based columns. ```yaml tables: users: converters: # Convert to lowercase email: converter: 'toLower' # Convert to uppercase country_code: converter: 'toUpper' ``` -------------------------------- ### Truncate Quote Tables for Performance Source: https://github.com/smile-sa/gdpr-dump/wiki/Guidelines Enable table truncation for quote tables to improve dump performance in Magento 2. ```yaml tables: quote: truncate: true ``` -------------------------------- ### Unset a Specific Converter by Setting to Null Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Remove a specific converter from a table's configuration by setting its key to `null`. This is only permitted if the converter is defined in a parent configuration. ```yaml extends: 'magento2' tables: admin_user: converters: email: ~ ``` -------------------------------- ### Declare Data Converter for Column Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Assign a data converter to a specific column in a table. Only non-null values are converted. ```yaml tables: my_table: converters: my_column: converter: 'randomizeText' ``` -------------------------------- ### Extend Magento 2 Template Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Specify the `magento2` template and its version when using it. This provides default anonymization rules for Magento 2 applications. ```yaml extends: 'magento2' version: '2.4.8' ``` -------------------------------- ### Share Anonymized Emails Between Tables Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Use the `cache_key` parameter to ensure the same anonymized value is generated for a field across different tables. Ensure `unique` is set consistently if used. ```yaml tables: customer_entity: converters: email: converter: 'randomizeEmail' cache_key: 'customer_email' unique: true newsletter_subscriber: converters: subscriber_email: converter: 'randomizeEmail' cache_key: 'customer_email' unique: true ``` -------------------------------- ### Random Email Generator Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Generates a random email address using a list of domains and random text for the username part. ```APIDOC ## Random Email Generator ### Description Generates a random email address. The username part of the email is generated with the `randomText` converter. ### Parameters #### Query Parameters - **domains** (array of strings) - Optional - A list of email domains. Defaults to ['example.com', 'example.net', 'example.org']. - **min_length** (integer) - Optional - The minimum length of the username. Defaults to 3. - **max_length** (integer) - Optional - The maximum length of the username. Defaults to 16. - **characters** (string) - Optional - A string that contains the characters used to generate the username. (Link to source for default characters) ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'randomEmail' parameters: domains: ['test.com'] min_length: 5 max_length: 20 ``` ``` -------------------------------- ### Replace string converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Replaces all occurrences of a search string with a replacement string. ```yaml tables: my_table: converters: my_column: converter: 'replace' parameters: search: 'bar' replacement: 'baz' ``` -------------------------------- ### Table Data Filtering: Row Limit Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Configure a maximum number of rows to include for a specific table. ```yaml tables: # Limit number of rows large_table: limit: 10000 ``` -------------------------------- ### Anonymize DateTime Configuration Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Similar to anonymizeDate, this converter anonymizes datetime values. The default format is 'Y-m-d H:i:s'. Ensure the input datetime format matches the specified 'format' parameter. ```yaml tables: my_table: converters: my_column: converter: 'anonymizeDateTime' ``` -------------------------------- ### Generate Unique Values Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Set the unique flag to true to ensure generated values do not contain duplicates. ```yaml tables: users: converters: # Ensure unique emails email: converter: 'randomizeEmail' unique: true # Ensure unique usernames username: converter: 'randomizeText' unique: true ``` -------------------------------- ### Configure serializedData Converter Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Anonymize specific fields within PHP serialized column data. ```yaml tables: legacy_users: converters: # Serialized PHP array data profile_data: converter: 'serializedData' parameters: converters: email: converter: 'anonymizeEmail' phone: converter: 'randomizeNumber' address: converter: 'anonymizeText' ``` -------------------------------- ### Anonymize Text Converter Usage Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Replaces characters with asterisks while preserving the first letter of each word. Supports custom replacement characters, minimum word length, and custom word delimiters. ```yaml tables: users: converters: full_name: converter: 'anonymizeText' address: converter: 'anonymizeText' parameters: replacement: '#' min_word_length: 3 username: converter: 'anonymizeText' parameters: delimiters: [' ', '_', '-', '.'] ``` -------------------------------- ### Set Faker Locale Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Change the default locale used by faker formatters by setting the `faker.locale` parameter. Note that the default phar distribution only includes 'en_US'. ```yaml faker: locale: 'de_DE' ``` -------------------------------- ### Set static value converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Assigns a fixed scalar value or null to a column. ```yaml tables: my_table: converters: my_column: converter: 'setValue' parameters: value: 0 ``` -------------------------------- ### Filter Data in Table Dump Source: https://github.com/smile-sa/gdpr-dump/wiki/Configuration-File Apply a WHERE condition to filter data included in the dump for a specific table. ```yaml tables: my_table: where: 'email like "%@acme.com" or created_at > date_sub(now(), interval 55 day)' ``` -------------------------------- ### Configure JsonData Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Anonymize specific fields within a JSON object using dot notation paths. ```yaml tables: my_table: converters: my_column: converter: 'jsonData' parameters: converters: customer.email: converter: 'anonymizeEmail' customer.username: converter: 'anonymizeText' ``` -------------------------------- ### Anonymize DateTime Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Similar to anonymizeDate, but with a default format for date and time. ```APIDOC ## Anonymize DateTime Converter ### Description Same as `anonymizeDate`, but the default value of the format parameter is `Y-m-d H:i:s` instead of `Y-m-d`. ### Parameters #### Query Parameters - **format** (string) - Optional - The date format. Defaults to 'Y-m-d H:i:s'. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'anonymizeDateTime' parameters: format: 'Y-m-d H:i:s' ``` ``` -------------------------------- ### Random Email Generator Configuration Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Use the randomEmail converter to generate random email addresses. It utilizes the randomText converter for the username part. You can specify a list of domains and control the length and characters used for the username. ```yaml tables: my_table: converters: my_column: converter: 'randomEmail' ``` -------------------------------- ### Filter Propagation: Disable Source: https://context7.com/smile-sa/gdpr-dump/llms.txt Completely disable the automatic propagation of filters through foreign key relationships. ```yaml filter_propagation: enabled: false ``` -------------------------------- ### Random DateTime Generator Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Generates a random date and time within a specified year range and format. ```APIDOC ## Random DateTime Generator ### Description Same as `randomDate`, but the default value of the format parameter is `Y-m-d H:i:s` instead of `Y-m-d`. ### Parameters #### Query Parameters - **format** (string) - Optional - The date format. Defaults to 'Y-m-d H:i:s'. - **min_year** (integer) - Optional - The minimum year. If set to `null`, the min year is the current year. Defaults to 1900. - **max_year** (integer) - Optional - The maximum year. If set to `null`, the max year is the current year. Defaults to null. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'randomDateTime' parameters: format: 'Y-m-d H:i:s' min_year: 2000 max_year: 2050 ``` ``` -------------------------------- ### Anonymize Date Converter Source: https://github.com/smile-sa/gdpr-dump/wiki/Data-Converters Anonymizes date values by randomizing the day and month while keeping the year unchanged. The input date format must match the specified format. ```APIDOC ## Anonymize Date Converter ### Description Anonymizes date values. It can be used to anonymize a date of birth. The day and month are randomized. The year is not changed. For example, one of the possible conversions for "1990-01-01" is "1990-11-25". The date format of the input value MUST match the `format` parameter, otherwise an exception is thrown. ### Parameters #### Query Parameters - **format** (string) - Optional - The date format. Defaults to 'Y-m-d'. ### Request Example ```yaml tables: my_table: converters: my_column: converter: 'anonymizeDate' parameters: format: 'Y-m-d' ``` ```