### Install Dependencies Source: https://github.com/qovery/replibyte/blob/main/website/README.md Run this command to install the project dependencies. ```bash $ yarn ``` -------------------------------- ### Start Local Development Server Source: https://github.com/qovery/replibyte/blob/main/website/README.md Starts a local development server with live reloading for changes. ```bash $ yarn start ``` -------------------------------- ### Install RepliByte from Source Source: https://github.com/qovery/replibyte/blob/main/website/docs/getting-started/installation.mdx Clone the repository and build the binary using Cargo. ```shell git clone https://github.com/Qovery/replibyte.git && cd replibyte # Install cargo # visit: https://doc.rust-lang.org/cargo/getting-started/installation.html # Build with cargo cargo build --release # Run RepliByte ./target/release/replibyte -h ``` -------------------------------- ### PostgreSQL Connection URI Examples Source: https://context7.com/qovery/replibyte/llms.txt Provides examples of connection URIs for PostgreSQL databases, including standard format and using environment variables. ```yaml source: connection_uri: postgres://user:password@host:port/database ``` ```yaml source: connection_uri: postgresql://user:password@host:port/database ``` ```yaml source: connection_uri: $DATABASE_URL ``` -------------------------------- ### Install Replibyte from Source Source: https://context7.com/qovery/replibyte/llms.txt Build the Replibyte binary from the source code using Cargo. ```shell git clone https://github.com/Qovery/replibyte.git && cd replibyte # Install cargo (https://doc.rust-lang.org/cargo/getting-started/installation.html) # Build with cargo cargo build --release # Run RepliByte ./target/release/replibyte -h ``` -------------------------------- ### Complete conf.yaml for Transformed Dump Creation Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md This is a comprehensive `conf.yaml` example for creating a transformed dump. It includes source connection details, transformer configurations for specific tables and columns, and an AWS S3 datastore setup. Remember to update the bucket name and credentials. ```yaml source: connection_uri: postgres://user:password@host:port/db # optional - use only for option #1 transformers: - database: public table: customers columns: - name: first_name transformer_name: first-name - name: last_name transformer_name: random - name: contact_phone transformer_name: phone-number - name: contact_email transformer_name: email datastore: aws: bucket: my-replibyte-dumps region: us-east-2 credentials: access_key_id: $ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY session_token: XXX # optional ``` -------------------------------- ### Install RepliByte on MacOSX Source: https://github.com/qovery/replibyte/blob/main/website/docs/getting-started/installation.mdx Use Homebrew to install RepliByte on MacOS systems. ```shell brew tap Qovery/replibyte brew install replibyte ``` -------------------------------- ### Install RepliByte on Linux Source: https://github.com/qovery/replibyte/blob/main/website/docs/getting-started/installation.mdx Download, extract, and move the RepliByte binary to a system-wide executable path. ```shell # download latest replibyte archive for Linux curl -s https://api.github.com/repos/Qovery/replibyte/releases/latest | \ jq -r '.assets[].browser_download_url' | \ grep -i 'linux-musl.tar.gz$' | wget -qi - && \ # unarchive tar zxf *.tar.gz # make replibyte executable chmod +x replibyte # make it accessible from everywhere mv replibyte /usr/local/bin/ ``` -------------------------------- ### MySQL Connection URI Example Source: https://context7.com/qovery/replibyte/llms.txt Shows the connection URI format for a MySQL database. ```yaml source: connection_uri: mysql://user:password@host:port/database ``` -------------------------------- ### Run Replibyte Container Locally Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/deploy-replibyte/container.md Starts the Replibyte container, mounting the local 'replibyte.yaml' configuration file and loading environment variables from 'env.txt'. ```sh docker run -it --name replibyte \ --env-file env.txt \ -v "$(pwd)/replibyte.yaml":/replibyte.yaml:ro \ ghcr.io/qovery/replibyte \ ``` -------------------------------- ### MongoDB Connection URI Examples Source: https://context7.com/qovery/replibyte/llms.txt Provides examples of connection URIs for MongoDB databases, including standard format and SRV records. ```yaml source: connection_uri: mongodb://user:password@host:port/database?authSource=admin ``` ```yaml source: connection_uri: mongodb+srv://user:password@server.example.com/database ``` -------------------------------- ### Local Disk Datastore Configuration Source: https://context7.com/qovery/replibyte/llms.txt Configuration example for using the local file system as a datastore. Specify the directory where dumps will be stored. ```yaml datastore: local_disk: dir: /data/replibyte ``` -------------------------------- ### Rust WebAssembly Transformer Example Source: https://context7.com/qovery/replibyte/llms.txt Example of creating a custom transformer using Rust and compiling it to WebAssembly. The code reads from stdin, transforms the input (reversing the string in this case), and writes to stdout. ```rust src/main.rs fn main() { // Read input value from stdin let mut input = String::new(); std::io::stdin().read_line(&mut input).unwrap(); // Transform the value (example: reverse string) let output: String = input.chars().rev().collect(); // Write transformed value to stdout println!("{}", output); } ``` -------------------------------- ### Initialize Rust Cargo Project Source: https://github.com/qovery/replibyte/blob/main/website/docs/advanced-guides/web-assembly-transformer.md Use this command to start a new Rust project for your custom WASM transformer. ```shell cargo init my-custom-wasm-transformer ``` -------------------------------- ### S3-Compatible Datastore Configuration (MinIO, DigitalOcean Spaces) Source: https://context7.com/qovery/replibyte/llms.txt Configuration example for S3-compatible object storage services like MinIO or DigitalOcean Spaces. Requires bucket, region, credentials, and a custom endpoint. ```yaml datastore: aws: bucket: my-bucket region: us-east-1 credentials: access_key_id: minioadmin secret_access_key: minioadmin endpoint: custom: 'http://localhost:9000' ``` -------------------------------- ### Create Directory for Local Disk Datastore Source: https://github.com/qovery/replibyte/blob/main/website/docs/datastores.mdx Example command to create a directory on your local disk for Replibyte to use as a datastore. The directory must be readable and writable by the user running Replibyte. ```sh mkdir /data/replibyte ``` -------------------------------- ### Configure Database Subsetting in conf.yaml Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/3-subset-a-dump.md Use the `database_subset` object in your `conf.yaml` to define subsetting parameters. This example configures subsetting for the `customers` table in the `public` database, using a random strategy to keep 10% of the data, and specifies `product_catalog` to be passed through entirely. ```yaml source: connection_uri: postgres://user:password@host:port/db transformers: - database: public table: customers columns: - name: first_name transformer_name: first-name - name: last_name transformer_name: random - name: contact_phone transformer_name: phone-number - name: contact_email transformer_name: email database_subset: database: public table: customers strategy_name: random strategy_options: percent: 10 passthrough_tables: - product_catalog ``` -------------------------------- ### AWS S3 Datastore Configuration Source: https://context7.com/qovery/replibyte/llms.txt Configuration example for using AWS S3 as a datastore. Includes bucket, region, and optional profile or credentials. ```yaml datastore: aws: bucket: my-replibyte-bucket region: us-east-2 profile: my-replibyte-bucket # optional - use AWS CLI profile credentials: # optional if using default AWS credentials access_key_id: $AWS_ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY session_token: $AWS_SESSION_TOKEN # optional ``` -------------------------------- ### Configure PostgreSQL Connection Source: https://github.com/qovery/replibyte/blob/main/website/docs/databases.mdx Use a postgres:// URI for source and destination connections. Requires pg_dump installed locally for remote operations. ```yaml source: connection_uri: postgres://:@:/ # you can use $DATABASE_URL #... destination: connection_uri: postgres://:@:/ # you can use $DATABASE_URL ``` -------------------------------- ### Configure MySQL or MariaDB Connection Source: https://github.com/qovery/replibyte/blob/main/website/docs/databases.mdx Use a mysql:// URI for source and destination connections. Requires mysqldump installed locally for remote operations. ```yaml source: connection_uri: mysql://:@:/ # you can use $DATABASE_URL #... destination: connection_uri: mysql://:@:/ # you can use $DATABASE_URL ``` -------------------------------- ### Configure Transformers for Sensitive Data in conf.yaml Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Example `conf.yaml` configuration demonstrating how to use transformers to anonymize sensitive columns like `first_name`, `last_name`, `contact_email`, and `contact_phone` in a `customers` table. ```yaml source: connection_uri: postgres://user:password@host:port/db transformers: - database: public table: customers columns: - name: first_name transformer_name: first-name - name: last_name transformer_name: random - name: contact_phone transformer_name: phone-number - name: contact_email transformer_name: email ``` -------------------------------- ### GCP Cloud Storage Datastore Configuration Source: https://context7.com/qovery/replibyte/llms.txt Configuration example for using Google Cloud Storage as a datastore. Requires bucket, region, and access keys. ```yaml datastore: gcp: bucket: my-gcp-bucket region: us-central1 access_key: $GS_ACCESS_KEY secret: $GS_SECRET ``` -------------------------------- ### SQL Data Masking Example Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Demonstrates masking sensitive data in a SQL INSERT statement. The original value is replaced with a masked version. ```sql INSERT INTO public.my_table (payment_card) VALUE ('1234 1234 1234 1234'); ``` ```sql INSERT INTO public.my_table (payment_card) VALUE ('123####################'); ``` -------------------------------- ### Configure MongoDB Connection Source: https://github.com/qovery/replibyte/blob/main/website/docs/databases.mdx Use a mongodb:// URI for source and destination connections. Requires mongodump installed locally for remote operations. ```yaml source: connection_uri: mongodb://:@:/? # you can use $DATABASE_URL #... destination: connection_uri: mongodb://:@:/? # you can use $DATABASE_URL ``` -------------------------------- ### Database Subsetting Configuration Source: https://context7.com/qovery/replibyte/llms.txt Configure database subsetting to create a smaller, consistent dataset. Specify the 'database', 'table' to start from, 'strategy_name', and 'strategy_options'. 'passthrough_tables' lists tables to be copied entirely. ```yaml source: connection_uri: postgres://user:password@host:port/db database_subset: database: public table: orders # Start subsetting from this table strategy_name: random strategy_options: percent: 10 # Keep approximately 10% of the data passthrough_tables: # Tables to copy entirely without subsetting - product_catalog - categories - settings transformers: - database: public table: customers columns: - name: email transformer_name: email - name: phone transformer_name: phone-number datastore: aws: bucket: my-bucket region: us-east-2 credentials: access_key_id: $AWS_ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY ``` -------------------------------- ### Email Transformer Configuration Source: https://context7.com/qovery/replibyte/llms.txt Example configuration for the 'email' transformer, which replaces email addresses with fake ones. This is applied to the 'contact_email' column in the 'customers' table. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: customers columns: - name: contact_email transformer_name: email ``` -------------------------------- ### Random Transformer Configuration Source: https://context7.com/qovery/replibyte/llms.txt Example configuration for the 'random' transformer, which randomizes string values while preserving their length. This is applied to the 'description' column in the 'users' table. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: users columns: - name: description transformer_name: random # Input: INSERT INTO users (description) VALUE ('Hello World'); # Output: INSERT INTO users (description) VALUE ('Awdka Qdkqd'); ``` -------------------------------- ### First Name Transformer Configuration Source: https://context7.com/qovery/replibyte/llms.txt Example configuration for the 'first-name' transformer, which replaces values with fake first names. This is applied to the 'first_name' column in the 'employees' table. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: employees columns: - name: first_name transformer_name: first-name # Input: INSERT INTO employees (first_name) VALUE ('Lucas'); # Output: INSERT INTO employees (first_name) VALUE ('Georges'); ``` -------------------------------- ### Restore Latest Dump Locally Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/2-restore-a-dump.md Use this command to restore the latest available dump to a local PostgreSQL instance. Ensure Docker is installed and running. The `-d` parameter can be changed to other supported database types like `mongodb` or `mysql`. ```shell replibyte -c conf.yaml dump restore local -d postgresql -v latest ``` -------------------------------- ### MongoDB Nested Object Transformation Configuration Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Configuration for transforming nested fields within a MongoDB document. This example targets 'email' and 'phone_number' within the 'contact' object. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: my_database table: my_collection columns: - name: contact.email transformer_name: email - name: contact.phone_number transformer_name: phone-number ``` -------------------------------- ### MongoDB Nested Array of Objects Transformation Configuration Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Configuration for transforming fields within an array of objects in a MongoDB document. This example targets 'email' and 'phone_number' within each object in the 'contacts' array. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: my_database table: my_collection columns: - name: contacts.$[].email transformer_name: email - name: contacts.$[].phone_number transformer_name: phone-number ``` -------------------------------- ### Run Replibyte with Configuration Source: https://github.com/qovery/replibyte/blob/main/website/docs/getting-started/configuration.md Execute the Replibyte application using a specified configuration file. ```shell replibyte -c conf.yaml ``` -------------------------------- ### List Available Dumps Source: https://context7.com/qovery/replibyte/llms.txt View all database dumps stored in the configured datastore. ```shell replibyte -c conf.yaml dump list # Output example: # name size when compressed encrypted # dump-1647706359405 154MB Yesterday at 03:00 am true true # dump-1647731334517 152MB 2 days ago at 03:00 am true true # dump-1647734369306 149MB 3 days ago at 03:00 am true true ``` -------------------------------- ### Run database backup from dump file Source: https://github.com/qovery/replibyte/blob/main/website/docs/faq.md Commands to execute a database backup using a dump file, either via standard input or a file path. ```shell cat dump.sql | replibyte -c conf.yaml backup run -s postgres -i ``` ```shell replibyte -c conf.yaml backup run -s postgres -f dump.sql ``` -------------------------------- ### Create a Database Dump with Replibyte Source: https://github.com/qovery/replibyte/blob/main/README.md Use this command to create a data dump from your database. Ensure your configuration file (conf.yaml) is correctly set up. ```shell replibyte -c conf.yaml dump create ``` -------------------------------- ### Create Database Dumps Source: https://context7.com/qovery/replibyte/llms.txt Generate database dumps from source databases or stdin with optional transformations. ```shell # Create dump directly from source database replibyte -c conf.yaml dump create # Create dump from stdin (useful for existing dump files) cat your_dump.sql | replibyte -c conf.yaml dump create -i -s postgresql # Create dump with custom name replibyte -c conf.yaml dump create --name my-custom-dump # Source type options: postgresql, mysql, mongodb cat mongo_dump.bson | replibyte -c conf.yaml dump create -i -s mongodb ``` -------------------------------- ### Build Static Content Source: https://github.com/qovery/replibyte/blob/main/website/README.md Generates static files in the build directory for production hosting. ```bash $ yarn build ``` -------------------------------- ### Run all local tests Source: https://github.com/qovery/replibyte/blob/main/website/docs/contributing.md Executes the full suite of tests to verify the local development environment. ```shell cargo test --all ``` -------------------------------- ### Visualize PostgreSQL Source Replication Flow Source: https://github.com/qovery/replibyte/blob/main/docs/DESIGN.md Sequence diagram illustrating the data extraction, transformation, and upload process from a source database to an S3 bridge. ```mermaid sequenceDiagram participant RepliByte participant PostgreSQL (Source) participant AWS S3 (Bridge) PostgreSQL (Source)->>RepliByte: 1. Dump data loop RepliByte->>RepliByte: 2. Subsetting (optional) RepliByte->>RepliByte: 3. Hide or fake sensitive data (optional) RepliByte->>RepliByte: 4. Compress data (optional) RepliByte->>RepliByte: 5. Encrypt data (optional) end RepliByte->>AWS S3 (Bridge): 6. Upload obfuscated dump data RepliByte->>AWS S3 (Bridge): 7. Write index file ``` -------------------------------- ### List Available Database Dumps Source: https://github.com/qovery/replibyte/blob/main/README.md This command lists all previously created dumps. It shows details like type, name, size, creation time, compression, and encryption status. ```shell replibyte -c conf.yaml dump list ``` -------------------------------- ### Deploy to GitHub Pages Source: https://github.com/qovery/replibyte/blob/main/website/README.md Commands to deploy the website to the gh-pages branch, with or without SSH authentication. ```bash $ USE_SSH=true yarn deploy ``` ```bash $ GIT_USER= yarn deploy ``` -------------------------------- ### List Available Transformers Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Command to display all built-in transformers and their descriptions. ```shell replibyte -c conf.yaml transformer list ``` -------------------------------- ### Configure Replibyte Source: https://context7.com/qovery/replibyte/llms.txt Define source database, datastore, transformations, and encryption settings in the YAML configuration file. ```yaml # Complete configuration example with all options encryption_key: $MY_PRIVATE_ENC_KEY # optional - encrypt data on datastore source: connection_uri: postgres://user:password@host:port/db # supports $DATABASE_URL env var # Optional: Subset large databases to a smaller size database_subset: database: public table: orders strategy_name: random strategy_options: percent: 50 passthrough_tables: - us_states # Tables to copy entirely without subsetting # Optional: Transform sensitive data columns transformers: - database: public table: employees columns: - name: last_name transformer_name: random - name: birth_date transformer_name: random-date - name: first_name transformer_name: first-name - name: email transformer_name: email - name: username transformer_name: keep-first-char - database: public table: customers columns: - name: phone transformer_name: phone-number # Optional: Dump only specific tables only_tables: - database: public table: orders - database: public table: customers # Optional: Skip specific tables from dump skip: - database: public table: audit_logs # Datastore configuration (choose one) datastore: aws: bucket: $BUCKET_NAME region: $S3_REGION credentials: access_key_id: $ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY session_token: XXX # optional destination: connection_uri: postgres://user:password@host:port/db wipe_database: true # default: true - wipes public schema before restore ``` -------------------------------- ### Run RepliByte with Docker Source: https://github.com/qovery/replibyte/blob/main/website/docs/getting-started/installation.mdx Execute RepliByte using a Docker container with a mounted configuration file. ```shell docker run -it --rm --name replibyte \ -v "$(pwd)/replibyte.yaml:/replibyte.yaml:ro" \ ghcr.io/qovery/replibyte --config replibyte.yaml ``` ```yaml datastore: local_disk: dir: ./my-datastore ``` -------------------------------- ### Create Transformed Dump from SQL File (Option 2 & 3) Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md This command creates a transformed dump by piping a SQL file into Replibyte. The `-i` flag is necessary for reading from standard input, and `-s` specifies the source database type if not defined in `conf.yaml`. ```shell cat your_dump.sql | replibyte -c conf.yaml dump create -i -s postgresql ``` -------------------------------- ### Dump Production Database with Replibyte Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/deploy-replibyte/container.md Runs the Replibyte container to create a dump of the production database, excluding sensitive data, and uploads it to S3. Ensure all environment variables are correctly set. ```bash docker run -e S3_ACCESS_KEY_ID=XXX \ -e S3_SECRET_ACCESS_KEY=YYY \ -e S3_REGION=us-east-2 \ -e S3_BUCKET=my-test-bucket \ -e SOURCE_CONNECTION_URI=postgres://... \ -e DESTINATION_CONNECTION_URI=postgres://... \ -e ENCRYPTION_SECRET=itIsASecret \ ghcr.io/qovery/replibyte replibyte dump create ``` -------------------------------- ### Configure Replibyte to Create MySQL Dump Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Use this configuration in your `conf.yaml` to have Replibyte create a dump from a MySQL database. Ensure the connection URI is correctly formatted. ```yaml source: connection_uri: mysql://[user]:[password]@[host]:[port]/[database] ``` -------------------------------- ### Configure Replibyte to Create PostgreSQL Dump Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Use this configuration in your `conf.yaml` to have Replibyte create a dump from a PostgreSQL database. Ensure the connection URI is correctly formatted. ```yaml source: connection_uri: postgres://[user]:[password]@[host]:[port]/[database] ``` -------------------------------- ### YAML Configuration for Replibyte Source: https://github.com/qovery/replibyte/blob/main/website/docs/getting-started/configuration.md Define your database connection, encryption key, and datastore settings in a conf.yaml file. Environment variables can be used for sensitive information. ```yaml encryption_key: $MY_PRIVATE_ENC_KEY # optional - encrypt data on datastore source: connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL datastore: aws: bucket: $BUCKET_NAME region: $S3_REGION credentials: access_key_id: $ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY destination: connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL ``` -------------------------------- ### Manually Create PostgreSQL Dump Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Command to create a dump from a PostgreSQL database manually. Options like `--column-inserts` and `--no-owner` are used. ```bash pg_dump --column-inserts --no-owner -h [host] -p [port] -U [username] [database] ``` -------------------------------- ### Run Replibyte with Docker Source: https://context7.com/qovery/replibyte/llms.txt Execute Replibyte inside a Docker container using a local configuration file. ```shell docker run -it --rm --name replibyte \ -v "$(pwd)/replibyte.yaml:/replibyte.yaml:ro" \ ghcr.io/qovery/replibyte --config replibyte.yaml ``` -------------------------------- ### Configure Replibyte YAML Source: https://github.com/qovery/replibyte/blob/main/website/docs/getting-started/configuration.md Defines the source database, transformation rules for sensitive data, and destination storage settings. ```yaml encryption_key: $MY_PRIVATE_ENC_KEY # optional - encrypt data on datastore source: connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL database_subset: # optional - downscale database while keeping it consistent database: public table: orders strategy_name: random strategy_options: percent: 50 passthrough_tables: - us_states transformers: # optional - hide sensitive data - database: public table: employees columns: - name: last_name transformer_name: random - name: birth_date transformer_name: random-date - name: first_name transformer_name: first-name - name: email transformer_name: email - name: username transformer_name: keep-first-char - database: public table: customers columns: - name: phone transformer_name: phone-number only_tables: # optional - dumps only specified tables. - database: public table: orders - database: public table: customers datastore: aws: bucket: $BUCKET_NAME region: $S3_REGION credentials: access_key_id: $ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY destination: connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL ``` -------------------------------- ### Configure Replibyte to Use Environment Variable for Connection URI Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Alternatively, you can specify the database connection URI using an environment variable in your `conf.yaml`. ```yaml source: connection_uri: $DATABASE_URL ``` -------------------------------- ### View Database Schema Source: https://context7.com/qovery/replibyte/llms.txt Displays the schema of the source database configured in `conf.yaml`. This command is useful for understanding the structure of the data before performing operations. ```shell replibyte -c conf.yaml source schema ``` -------------------------------- ### Seed Development Database with Replibyte Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/deploy-replibyte/container.md Restores the latest production dump from S3 to a development database using the Replibyte container. Requires correct environment variable configuration. ```bash docker run -e S3_ACCESS_KEY_ID=XXX \ -e S3_SECRET_ACCESS_KEY=YYY \ -e S3_REGION=us-east-2 \ -e S3_BUCKET=my-test-bucket \ -e SOURCE_CONNECTION_URI=postgres://... \ -e DESTINATION_CONNECTION_URI=postgres://... \ -e ENCRYPTION_SECRET=itIsASecret \ ghcr.io/qovery/replibyte replibyte dump restore remote -v latest ``` -------------------------------- ### Run RepliByte tests with environment variables Source: https://github.com/qovery/replibyte/blob/main/website/docs/contributing.md Executes tests using specific AWS credentials for the local MinIO instance. ```shell AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin cargo test ``` -------------------------------- ### Configure Replibyte to Create MongoDB Dump Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Use this configuration in your `conf.yaml` to have Replibyte create a dump from a MongoDB database. Ensure the connection URI is correctly formatted. ```yaml source: connection_uri: mongodb://[user]:[password]@[host]:[port]/[database] ``` -------------------------------- ### Restore Dump to File Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/2-restore-a-dump.md This command restores the latest dump to a specified file (e.g., `dump.sql`) instead of directly into a database. This is useful for inspecting the dump's content manually. Ensure the output file path is correct. ```shell replibyte -c conf.yaml dump restore local -i postgres -v latest -o > dump.sql ``` -------------------------------- ### Manually Create MySQL Dump Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Command to create a dump from a MySQL database manually. Includes options for creating drop statements and using single transactions. ```bash mysqldump -h [host] -P [port] -u [username] -p --add-drop-database --add-drop-table --skip-extended-insert --complete-insert --single-transaction --quick --databases ``` -------------------------------- ### Replibyte Local Disk Datastore Configuration Source: https://github.com/qovery/replibyte/blob/main/website/docs/datastores.mdx Configure Replibyte to use a local directory as a datastore. Ensure the specified directory path is readable and writable by the Replibyte process. ```yaml ... datastore: local_disk: dir: ... ``` ```yaml ... datastore: local_disk: dir: /data/replibyte ... ``` -------------------------------- ### List available dumps Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/4-delete-a-dump.md Retrieves a list of all existing dumps to identify names for deletion. ```shell replibyte -c conf.yaml dump list type name size when compressed encrypted PostgreSQL dump-1647706359405 154MB Yesterday at 03:00 am true true PostgreSQL dump-1647731334517 152MB 2 days ago at 03:00 am true true PostgreSQL dump-1647734369306 149MB 3 days ago at 03:00 am true true ``` -------------------------------- ### Build WebAssembly Module Source: https://context7.com/qovery/replibyte/llms.txt Command to build a Rust project into a WebAssembly module targeting wasm32-wasi. The output is typically found in the target/wasm32-wasi/release/ directory. ```shell rustup target add wasm32-wasi cargo build --release --target wasm32-wasi # Output: target/wasm32-wasi/release/my-transformer.wasm ``` -------------------------------- ### Download Replibyte Docker Image Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/deploy-replibyte/container.md Pulls the latest official Replibyte Docker image from GitHub Container Registry. Check the package for available tags. ```sh docker pull ghcr.io/qovery/replibyte ``` -------------------------------- ### Configure Remote Destination Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/2-restore-a-dump.md Configure the `conf.yaml` file to specify the connection URI for a remote database where the dump will be restored. The `wipe_database` option can be set to `false` to prevent wiping the destination database. ```yaml destination: connection_uri: postgres://user:password@host:port/db # Disable public's schema wipe # wipe_database: false (default: true) ``` -------------------------------- ### Environment Variables for Replibyte Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/deploy-replibyte/container.md Lists environment variables required for Replibyte configuration, including S3 credentials, bucket details, and database connection URIs. Secure variables can be left empty to read from the shell. ```sh $cat env.txt S3_ACCESS_KEY_ID S3_SECRET_ACCESS_KEY S3_REGION=us-east-2 S3_BUCKET=my-test-bucket SOURCE_CONNECTION_URI=postgres://... DESTINATION_CONNECTION_URI=postgres://... ENCRYPTION_SECRET ``` -------------------------------- ### Build WASM Transformer Source: https://github.com/qovery/replibyte/blob/main/website/docs/advanced-guides/web-assembly-transformer.md Compile your Rust project into a WebAssembly module using the specified target. ```shell cargo build --release --target wasm32-wasi ``` -------------------------------- ### Define Manifest Index File Structure Source: https://github.com/qovery/replibyte/blob/main/docs/DESIGN.md JSON schema for the manifest file located at the root of the target bridge, detailing dump metadata. ```json { "dumps": [ { "size": 1024000, "directory_name": "dump-{epoch timestamp}", "created_at": "epoch timestamp", "compressed": true, "encrypted": true } ] } ``` -------------------------------- ### Manually Create MongoDB Dump Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Command to create a dump from a MongoDB database manually. The `--archive` option is used for creating a single archive file. ```bash mongodump -h [host] --port [port] --authenticationDatabase [auth_db|default: admin] --db [database] -u [username] -p [password] --archive ``` -------------------------------- ### Restore Dumps to Local Docker Source: https://context7.com/qovery/replibyte/llms.txt Restore a specific or the latest database dump into a local Docker container. ```shell # Restore latest dump to local PostgreSQL container replibyte -c conf.yaml dump restore local -v latest -i postgres -p 5432 # Restore specific dump to local PostgreSQL with custom port replibyte -c conf.yaml dump restore local -v dump-1647731334517 -i postgres -p 5433 ``` -------------------------------- ### Replibyte Configuration File Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/deploy-replibyte/container.md Defines the Replibyte configuration, including encryption, source, datastore (S3), and destination connections. Ensure this file is named 'replibyte.yaml'. ```yaml encryption_key: $ENCRYPTION_SECRET source: connection_uri: $SOURCE_CONNECTION_URI transformers: - database: public table: customers columns: - name: first_name transformer_name: first-name - name: last_name transformer_name: random - name: contact_phone transformer_name: phone-number - name: contact_email transformer_name: email datastore: aws: bucket: $S3_BUCKET region: $S3_REGION access_key_id: $S3_ACCESS_KEY_ID secret_access_key: $S3_SECRET_ACCESS_KEY destination: connection_uri: $DESTINATION_CONNECTION_URI ``` -------------------------------- ### Visualize PostgreSQL Destination Restoration Flow Source: https://github.com/qovery/replibyte/blob/main/docs/DESIGN.md Sequence diagram illustrating the process of retrieving, decrypting, and restoring data from an S3 bridge to a destination database. ```mermaid sequenceDiagram participant RepliByte participant PostgreSQL (Destination) participant AWS S3 (Bridge) AWS S3 (Bridge)->>RepliByte: 1. Read index file AWS S3 (Bridge)->>RepliByte: 2. Download dump SQL file loop RepliByte->>RepliByte: 3. Decrypt data (if required) RepliByte->>RepliByte: 4. Uncompress data (if required) end RepliByte->>PostgreSQL (Destination): 5. Restore dump SQL ``` -------------------------------- ### Custom WebAssembly Transformer Configuration Source: https://context7.com/qovery/replibyte/llms.txt Integrate custom data transformations using WebAssembly modules. Specify the 'path' to your WASM file in the transformer options. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: users columns: - name: custom_field transformer_name: custom-wasm transformer_options: path: "path/to/custom-transformer.wasm" ``` -------------------------------- ### Configure AWS S3 Datastore in conf.yaml Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/1-create-a-dump.md Add this section to your `conf.yaml` to specify an AWS S3 bucket as a datastore for your transformed dumps. Ensure you replace 'my-replibyte-dumps' with your actual bucket name and provide valid AWS credentials. ```yaml datastore: aws: bucket: my-replibyte-dumps region: us-east-2 credentials: access_key_id: $ACCESS_KEY_ID secret_access_key: $AWS_SECRET_ACCESS_KEY session_token: XXX # optional ``` -------------------------------- ### Restore Dump and Remove Container on Ctrl-C Source: https://context7.com/qovery/replibyte/llms.txt Restores the latest dump to a local PostgreSQL container and configures the container to be removed automatically when the process is interrupted (e.g., by Ctrl-C). ```shell replibyte -c conf.yaml dump restore local -v latest -i postgres -p 5432 -r ``` -------------------------------- ### Configure RepliByte with Custom WASM Transformer Source: https://github.com/qovery/replibyte/blob/main/website/docs/advanced-guides/web-assembly-transformer.md Update your `replibyte.yaml` file to specify the path to your compiled custom WASM transformer. ```yaml # ... - database: table: columns: - name: transformer_name: custom-wasm transformer_options: path: "path/to/your/custom-wasm-transformer.wasm" # ... ``` -------------------------------- ### First Name Transformer Configuration and Usage Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Replaces column values with generated fake first names. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: my_table columns: - name: first_name transformer_name: first-name # ... ``` ```sql INSERT INTO public.my_table (first_name) VALUE ('Lucas'); ``` ```sql INSERT INTO public.my_table (first_name) VALUE ('Georges'); ``` -------------------------------- ### Output Dump to Stdout Instead of Docker Source: https://context7.com/qovery/replibyte/llms.txt Restores the latest dump and outputs the dump content directly to standard output instead of writing it to a Docker container. This is useful for piping the dump to other commands or files. ```shell replibyte -c conf.yaml dump restore local -v latest -o > dump.sql ``` -------------------------------- ### Email Transformer Configuration and Usage Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Replaces string values with fake email addresses. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: my_table columns: - name: contact_email transformer_name: email # ... ``` ```sql INSERT INTO public.my_table (contact_email) VALUE ('tony.stark@random.com'); ``` ```sql INSERT INTO public.my_table (contact_email) VALUE ('toto@domain.tld'); ``` -------------------------------- ### Restore Latest Dump to Local PostgreSQL Container Source: https://github.com/qovery/replibyte/blob/main/README.md Restores the latest database dump into a local PostgreSQL container. Specify the container port if it's not the default. ```shell replibyte -c conf.yaml dump restore local -v latest -i postgres -p 5432 ``` -------------------------------- ### Output Remote Restore to Stdout Source: https://context7.com/qovery/replibyte/llms.txt Restores a dump to a remote destination and outputs the dump content to standard output. This is useful for redirecting the dump to a file or processing it with other tools. ```shell replibyte -c conf.yaml dump restore remote -v latest -o > dump.sql ``` -------------------------------- ### Restore Dump with Custom Docker Image Tag Source: https://context7.com/qovery/replibyte/llms.txt Restores the latest dump to a local PostgreSQL container using a specific Docker image tag. This allows for version control of the container image used for restoration. ```shell replibyte -c conf.yaml dump restore local -v latest -i postgres -p 5432 -t 14-alpine ``` -------------------------------- ### Replibyte S3 Compatible Datastore Configuration with Custom Endpoint Source: https://github.com/qovery/replibyte/blob/main/website/docs/datastores.mdx Configure Replibyte to use an S3-compatible datastore by specifying a custom endpoint. HMAC keys are required for authentication. ```yaml ... datastore: aws: bucket: region: credentials: access_key_id: XXX secret_access_key: XXX endpoint: custom: 'https://your-s3-compatible-endpoint' ... ``` -------------------------------- ### Credit Card Transformer Configuration and Usage Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Generates a fake credit card number. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: my_table columns: - name: payment_card transformer_name: card-number # ... ``` ```sql INSERT INTO public.my_table (payment_card) VALUE ('1234123412341234'); ``` ```sql INSERT INTO public.my_table (payment_card) VALUE ('5678567856785678'); ``` -------------------------------- ### Restore Dump to Local MongoDB Container Source: https://context7.com/qovery/replibyte/llms.txt Restores the latest dump to a local MongoDB container. Ensure the MongoDB container is running and accessible on port 27017. ```shell replibyte -c conf.yaml dump restore local -v latest -i mongodb -p 27017 ``` -------------------------------- ### Restore Dump to Local MySQL Container Source: https://context7.com/qovery/replibyte/llms.txt Restores the latest dump to a local MySQL container. Ensure the MySQL container is running and accessible on port 3306. ```shell replibyte -c conf.yaml dump restore local -v latest -i mysql -p 3306 ``` -------------------------------- ### Keep a maximum number of dumps Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/4-delete-a-dump.md Retains only the most recent number of dumps specified. ```shell replibyte -c conf.yaml dump delete --keep-last=10 ``` -------------------------------- ### Rust WASM Transformer Implementation Source: https://github.com/qovery/replibyte/blob/main/website/docs/advanced-guides/web-assembly-transformer.md This Rust code reads a string from stdin, reverses it, and prints the result to stdout. It serves as the core logic for the WASM transformer. ```rust // This is actually the source of the `.wasm` file in this example. Feel free to edit it ! fn main() { // Read input value from stdin let mut input = String::new(); std::io::stdin().read_line(&mut input).unwrap(); // Transform the value as you see fit (in this case we just reverse the string) let output: String = input.chars().rev().collect(); // Write transformed value to stdout (simply print) println!("{}", output); } ``` -------------------------------- ### Replibyte AWS S3 Datastore Configuration Source: https://github.com/qovery/replibyte/blob/main/website/docs/datastores.mdx Configure Replibyte to use AWS S3 as a datastore. Optional fields like 'profile', 'region', and 'credentials' can be omitted to use default AWS configuration mechanisms. ```yaml ... datastore: aws: bucket: profile: # optional region: # optional credentials: # optional access_key_id: XXX secret_access_key: XXX session_token: XXX # optional ... ``` -------------------------------- ### Restore Specific Dump Locally Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/2-restore-a-dump.md Restore a particular dump by its name to a local PostgreSQL instance. Replace `dump-1647731334517` with the actual name of the dump you wish to restore. ```shell replibyte -c conf.yaml dump restore local -d postgres -v dump-1647731334517 ``` -------------------------------- ### Restore Latest Dump to Remote Database Source: https://github.com/qovery/replibyte/blob/main/README.md Restores the latest database dump to a remote database. This command assumes remote connection details are configured. ```shell replibyte -c conf.yaml dump restore remote -v latest ``` -------------------------------- ### Parse Postgres INSERT Statement Source: https://github.com/qovery/replibyte/blob/main/dump-parser/README.md Parses a Postgres INSERT statement to extract column values. Ensure the input string is a valid SQL INSERT query. ```rust let q = r"\nINSERT INTO public.customers (customer_id, company_name, contact_name, contact_title)\nVALUES (1, 'Alfreds Futterkiste', 'Maria Anders', NULL);\n"; let mut tokenizer = Tokenizer::new(q); let tokens_result = tokenizer.tokenize(); assert_eq!(tokens_result.is_ok(), true); let tokens = trim_pre_whitespaces(tokens_result.unwrap()); let column_values = get_column_values_from_insert_into_query(&tokens); assert_eq!( column_values, vec![ &Token::Number("1".to_string(), false), &Token::SingleQuotedString("Alfreds Futterkiste".to_string()), &Token::SingleQuotedString("Maria Anders".to_string()), &Token::make_keyword("NULL"), ] ); ``` -------------------------------- ### Add WASM Target for Rust Source: https://github.com/qovery/replibyte/blob/main/website/docs/advanced-guides/web-assembly-transformer.md Add the `wasm32-wasi` target to your Rust toolchain to enable cross-compilation for WebAssembly. ```shell rustup target add wasm32-wasi ``` -------------------------------- ### Delete Dumps Older Than Specified Duration Source: https://context7.com/qovery/replibyte/llms.txt Deletes all dumps from the datastore that are older than the specified duration (e.g., '14d' for 14 days). This helps in managing storage space by removing old backups. ```shell replibyte -c conf.yaml dump delete --older-than=14d ``` -------------------------------- ### Restore Specific Dump to Remote Destination Source: https://context7.com/qovery/replibyte/llms.txt Restores a specific dump identified by its unique name to a remote database destination. Use this when you need to restore a particular version of your data. ```shell replibyte -c conf.yaml dump restore remote -v dump-1647706359405 ``` -------------------------------- ### Delete Specific Dump by Name Source: https://context7.com/qovery/replibyte/llms.txt Deletes a specific dump from the datastore identified by its unique name. Ensure the name is correct before executing to avoid accidental data loss. ```shell replibyte -c conf.yaml dump delete dump-1647706359405 ``` -------------------------------- ### Random Transformer Configuration and Usage Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Randomizes string values while maintaining the original length. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: my_table columns: - name: description transformer_name: random # ... ``` ```sql INSERT INTO public.my_table (description) VALUE ('Hello World'); ``` ```sql INSERT INTO public.my_table (description) VALUE ('Awdka Qdkqd'); ``` -------------------------------- ### Credit Card Transformer Configuration Source: https://context7.com/qovery/replibyte/llms.txt Configure the credit-card transformer to generate fake credit card numbers. This transformer is applied to the 'card_number' column in the 'payments' table. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: payments columns: - name: card_number transformer_name: credit-card ``` -------------------------------- ### Phone Number Transformer Configuration and Usage Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Generates a fake US phone number. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: my_table columns: - name: contact_phone transformer_name: phone-number # ... ``` ```sql INSERT INTO public.my_table (contact_phone) VALUE ('+123456789'); ``` ```sql INSERT INTO public.my_table (contact_phone) VALUE ('+356433821'); ``` -------------------------------- ### Delete a dump by name Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/4-delete-a-dump.md Removes a specific dump identified by its name. ```shell replibyte -c conf.yaml dump delete ``` -------------------------------- ### Delete dumps by age Source: https://github.com/qovery/replibyte/blob/main/website/docs/guides/4-delete-a-dump.md Removes dumps older than a specified number of days. Only the 'd' unit is currently supported. ```shell replibyte -c conf.yaml dump delete --older-than=2d ``` -------------------------------- ### Replibyte GCP Cloud Storage Datastore Configuration Source: https://github.com/qovery/replibyte/blob/main/website/docs/datastores.mdx Configure Replibyte to use GCP Cloud Storage as a datastore. Ensure you have generated compatible S3 API keys and created the bucket manually. ```yaml ... datastore: gcp: bucket: your-bucket-name region: us-central1 access_key: $GS_ACCESS_KEY secret: $GS_SECRET ... ``` -------------------------------- ### Redacted Transformer Configuration and Usage Source: https://github.com/qovery/replibyte/blob/main/website/docs/transformers.md Obfuscates sensitive data with options for custom width and masking characters. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: my_table columns: - name: payment_card transformer_name: redacted # ... ``` ```sql INSERT INTO public.my_table (payment_card) VALUE ('1234 1234 1234 1234'); ``` ```sql INSERT INTO public.my_table (payment_card) VALUE ('1234***************'); ``` ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: employees columns: - name: last_name transformer_name: redacted transformer_options: character: '#' width: 20 ``` -------------------------------- ### MongoDB Nested Field Transformation Source: https://context7.com/qovery/replibyte/llms.txt Configure transformers for nested fields in MongoDB. Use dot notation for sub-documents (e.g., 'contact.email') and bracket notation for arrays (e.g., 'contacts.$[].email'). ```yaml # For embedded sub-documents # Document: { "contact": { "email": "john@example.com", "phone": "123456" } } source: connection_uri: $DATABASE_URL transformers: - database: my_database table: my_collection columns: - name: contact.email transformer_name: email - name: contact.phone transformer_name: phone-number ``` ```yaml # For arrays of embedded documents # Document: { "contacts": [{ "email": "john@example.com" }, { "email": "jane@example.com" }] } source: connection_uri: $DATABASE_URL transformers: - database: my_database table: my_collection columns: - name: contacts.$[].email transformer_name: email - name: contacts.$[].phone_number transformer_name: phone-number ``` -------------------------------- ### Phone Number Transformer Configuration Source: https://context7.com/qovery/replibyte/llms.txt Use the phone-number transformer to generate fake phone numbers for a specified column. Ensure the 'phone-number' transformer is listed under 'transformers'. ```yaml source: connection_uri: $DATABASE_URL transformers: - database: public table: contacts columns: - name: phone transformer_name: phone-number ```