Replibyte

https://www.replibyte.com/
Admin
Replibyte is a blazingly fast tool to seed databases with production data while keeping sensitive...
Tokens:9,832
Snippets:152
Trust Score:7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install RepliByte from Source

Source: https://www.replibyte.com/docs/getting-started/installation

Clone the repository and build the binary using Cargo.

```bash
git clone https://github.com/Qovery/replibyte.git && cd replibyte  
  
# Install cargo  
# visit: https://doc.rust-lang.org/cargo/getting-started/installation.html  
  
# Build with cargo  
cargo build --release  
  
# Run RepliByte  
./target/release/replibyte -h  
```

--------------------------------

### Install RepliByte on MacOSX

Source: https://www.replibyte.com/docs/getting-started/installation

Use Homebrew to tap and install the RepliByte binary on macOS.

```bash
brew tap Qovery/replibyte  
brew install replibyte  
```

--------------------------------

### Configure Local Disk Datastore

Source: https://www.replibyte.com/docs/getting-started/installation

Example configuration snippet for the local_disk datastore.

```yaml
datastore:  
  local_disk:  
    dir: ./my-datastore  
```

--------------------------------

### Install RepliByte on Linux

Source: https://www.replibyte.com/docs/getting-started/installation

Download, extract, and move the RepliByte binary to the system path.

```bash
# download latest replibyte archive for Linux  
curl -s https://api.github.com/repos/Qovery/replibyte/releases/latest | \
  jq -r '.assets[].browser_download_url' | \
  grep -i 'linux-musl.tar.gz$' | wget -qi - && \
  
# unarchive  
tar zxf *.tar.gz  
  
# make replibyte executable  
chmod +x replibyte  
  
# make it accessible from everywhere  
mv replibyte /usr/local/bin/  
```

--------------------------------

### Start local databases with Docker Compose

Source: https://www.replibyte.com/docs/contributing

Initializes the required PostgreSQL, MySQL, MongoDB, and MinIO instances for development.

```bash
docker compose -f ./docker-compose-dev.yml up
```

--------------------------------

### Configure First Name Transformer

Source: https://www.replibyte.com/docs/transformers

Configuration and SQL transformation example for the first-name transformer.

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: my_table  
      columns:  
        - name: first_name  
          transformer_name: first-name  
# ...  
```

```sql
INSERT INTO public.my_table (first_name) VALUE ('Lucas');  
```

```sql
INSERT INTO public.my_table (first_name) VALUE ('Georges');  
```

--------------------------------

### Start Replibyte container

Source: https://www.replibyte.com/docs/guides/deploy-replibyte/container

Run the Replibyte container locally using the configuration file and environment variables.

```bash
docker run -it --name replibyte \
    --env-file env.txt \
    -v "$(pwd)/replibyte.yaml":/replibyte.yaml:ro \
    ghcr.io/qovery/replibyte  
```

--------------------------------

### Configure Email Transformer

Source: https://www.replibyte.com/docs/transformers

Configuration and SQL transformation example for the email transformer.

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: my_table  
      columns:  
        - name: contact_email  
          transformer_name: email  
# ...  
```

```sql
INSERT INTO public.my_table (contact_email) VALUE ('tony.stark@random.com');  
```

```sql
INSERT INTO public.my_table (contact_email) VALUE ('toto@domain.tld');  
```

--------------------------------

### Initialize Rust Cargo Project

Source: https://www.replibyte.com/docs/advanced-guides/web-assembly-transformer

Use this command to start a new Rust project for your WASM transformer.

```bash
cargo init my-custom-wasm-transformer  

```

--------------------------------

### Configure Random Transformer

Source: https://www.replibyte.com/docs/transformers

Configuration and SQL transformation example for the random string transformer.

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: my_table  
      columns:  
        - name: description  
          transformer_name: random  
# ...  
```

```sql
INSERT INTO public.my_table (description) VALUE ('Hello World');  
```

```sql
INSERT INTO public.my_table (description) VALUE ('Awdka Qdkqd');  
```

--------------------------------

### Configure Credit Card Transformer

Source: https://www.replibyte.com/docs/transformers

Configuration and SQL transformation example for the card-number transformer.

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: my_table  
      columns:  
        - name: payment_card  
          transformer_name: card-number  
# ...  
```

```sql
INSERT INTO public.my_table (payment_card) VALUE ('1234123412341234');  
```

```sql
INSERT INTO public.my_table (payment_card) VALUE ('5678567856785678');  
```

--------------------------------

### Configure Phone Number Transformer

Source: https://www.replibyte.com/docs/transformers

Configuration and SQL transformation example for the phone-number transformer.

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: my_table  
      columns:  
        - name: contact_phone  
          transformer_name: phone-number  
# ...  
```

```sql
INSERT INTO public.my_table (contact_phone) VALUE ('+123456789');  
```

```sql
INSERT INTO public.my_table (contact_phone) VALUE ('+356433821');  
```

--------------------------------

### Configure Keep First Character Transformer

Source: https://www.replibyte.com/docs/transformers

Configuration and SQL transformation example for the keep-first-char transformer.

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: my_table  
      columns:  
        - name: first_name  
          transformer_name: keep-first-char  
# ...  
```

```sql
INSERT INTO public.my_table (first_name) VALUE ('Lucas');  
```

```sql
INSERT INTO public.my_table (first_name) VALUE ('L');  
```

--------------------------------

### Example Embedded Sub-document Structure

Source: https://www.replibyte.com/docs/transformers

An example JSON document structure with an embedded sub-document containing email and phone number.

```json
{
  "contact": {
      "email": "john.doe@example.com",
      "phone_number": "123456"
  }
}

```

--------------------------------

### Example Embedded Array of Documents Structure

Source: https://www.replibyte.com/docs/transformers

An example JSON document structure with an array of embedded documents, each containing email and phone number.

```json
{
  "contacts": [
    {
      "email": "john.doe@example.com",
      "phone_number": "123456"
    },
    {
      "email": "jane.doe@example.com",
      "phone_number": "123456"
    }
  ]
}

```

--------------------------------

### Configure PostgreSQL Connection URI

Source: https://www.replibyte.com/docs/databases

Use a prefixed connection URI with `postgres://` for PostgreSQL. Ensure `pg_dump` is installed for remote backup and restore operations.

```yaml
source:  
  connection_uri: postgres://<user>:<password>@<host>:<port>/<database> # you can use $DATABASE_URL  
#...  
destination:  
  connection_uri: postgres://<user>:<password>@<host>:<port>/<database> # you can use $DATABASE_URL  


```

--------------------------------

### Configure MySQL/MariaDB Connection URI

Source: https://www.replibyte.com/docs/databases

Use a prefixed connection URI with `mysql://` for MySQL or MariaDB. Ensure `mysqldump` is installed for remote backup and restore operations.

```yaml
source:  
  connection_uri: mysql://<user>:<password>@<host>:<port>/<database> # you can use $DATABASE_URL  
#...  
destination:  
  connection_uri: mysql://<user>:<password>@<host>:<port>/<database> # you can use $DATABASE_URL  


```

--------------------------------

### Define database schema and entries for transformation

Source: https://www.replibyte.com/docs/guides/create-a-dump

Example SQL schema and data entries used to demonstrate data masking.

```sql
CREATE TABLE public.customers (  
    id bpchar NOT NULL,  
    first_name character varying(30) NOT NULL,  
    last_name character varying(30) NOT NULL,  
    contact_email character varying(2048) NOT NULL,  
    contact_phone character varying(24)  
);  

```

```sql
INSERT INTO public.customers (id, first_name, last_name, contact_email, contact_phone) VALUES ('ALFKI', 'Maria', 'Anders', 'maria.anders@gmail.com', '030-0074321');  
INSERT INTO public.customers (id, first_name, last_name, contact_email, contact_phone) VALUES ('ANATR', 'Ana', 'Trujillo', 'ana@factchecker.com', '(5) 555-4729');  
INSERT INTO public.customers (id, first_name, last_name, contact_email, contact_phone) VALUES ('ANTON', 'Antonio', 'Moreno', 'anto.moreno@gmail.com', NULL);  

```

--------------------------------

### Configure Redacted Transformer

Source: https://www.replibyte.com/docs/transformers

Configuration and SQL transformation examples for the redacted transformer, including advanced options.

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: my_table  
      columns:  
        - name: payment_card  
          transformer_name: redacted  
# ...  
```

```sql
INSERT INTO public.my_table (payment_card) VALUE ('1234 1234 1234 1234');  
```

```sql
INSERT INTO public.my_table (payment_card) VALUE ('1234***************');  
```

```yaml
source:  
  connection_uri: $DATABASE_URL  
  transformers:  
    - database: public  
      table: employees  
      columns:  
        - name: last_name  
          transformer_name: redacted  
          transformer_options:  
            character: '#'  
            width: 20  
```

--------------------------------

### Configure MongoDB Connection URI

Source: https://www.replibyte.com/docs/databases

Use a prefixed connection URI with `mongodb://` for MongoDB. Ensure `mongodump` is installed for remote backup and restore operations.

```yaml
source:  
  connection_uri: mongodb://<user>:<password>@<host>:<port>/<database>?<options> # you can use $DATABASE_URL  
#...  
destination:  
  connection_uri: mongodb://<user>:<password>@<host>:<port>/<database>?<options> # you can use $DATABASE_URL  


```

--------------------------------

### Restore Latest Dump Locally with Docker

Source: https://www.replibyte.com/docs/guides/restore-a-dump

Use this command to restore the latest available dump into a local Docker instance. Ensure Docker is installed and running. The -d parameter accepts various database types.

```bash
replibyte -c conf.yaml dump restore local -d postgresql -v latest  

```

--------------------------------

### Run Replibyte

Source: https://www.replibyte.com/docs/getting-started/configuration

Executes the application using the specified configuration file.

```bash
replibyte -c conf.yaml  
```

--------------------------------

### Create a database dump

Source: https://www.replibyte.com/

Initiates a new database dump process using the specified configuration file.

```bash
replibyte -c conf.yaml dump create  
```

--------------------------------

### Run backup from a dump file

Source: https://www.replibyte.com/docs/faq

Commands to execute a database backup using a dump file as input.

```bash
cat dump.sql | replibyte -c conf.yaml backup run -s postgres -i  
```

```bash
replibyte -c conf.yaml backup run -s postgres -f dump.sql  
```

--------------------------------

### Advanced Configuration with Transformations

Source: https://www.replibyte.com/docs/getting-started/configuration

Demonstrates how to apply data transformers and database subsetting to mask sensitive information and reduce dataset size.

```yaml
encryption_key: $MY_PRIVATE_ENC_KEY # optional - encrypt data on datastore  
source:  
  connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL  
  database_subset: # optional - downscale database while keeping it consistent  
    database: public  
    table: orders  
    strategy_name: random  
    strategy_options:  
      percent: 50  
    passthrough_tables:  
      - us_states  
  transformers: # optional - hide sensitive data  
    - database: public  
      table: employees  
      columns:  
        - name: last_name  
          transformer_name: random  
        - name: birth_date  
          transformer_name: random-date  
        - name: first_name  
          transformer_name: first-name  
        - name: email  
          transformer_name: email  
        - name: username  
          transformer_name: keep-first-char  
    - database: public  
      table: customers  
      columns:  
        - name: phone  
          transformer_name: phone-number  
  only_tables: # optional - dumps only specified tables.  
    - database: public  
      table: orders  
    - database: public  
      table: customers  
datastore:  
  aws:  
    bucket: $BUCKET_NAME  
    region: $S3_REGION  
    credentials:  
      access_key_id: $ACCESS_KEY_ID  
      secret_access_key: $AWS_SECRET_ACCESS_KEY  
destination:  
  connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL  
```

--------------------------------

### Run all local tests

Source: https://www.replibyte.com/docs/contributing

Verifies the development environment by executing the full test suite.

```bash
cargo test --all
```

--------------------------------

### Replibyte Dump Commands

Source: https://www.replibyte.com/docs/guides/create-a-dump

Commands to initiate a database dump using the provided configuration file.

```bash
replibyte -c conf.yaml dump create  
```

```bash
cat your_dump.sql | replibyte -c conf.yaml dump create -i -s postgresql  
```

--------------------------------

### List Available Dumps

Source: https://www.replibyte.com/docs/guides/restore-a-dump

Execute this command to see a list of all available dumps, including their type, name, size, and when they were created. This helps in choosing which specific dump to restore.

```bash
replibyte -c conf.yaml dump list  

```

--------------------------------

### Create Local Storage Directory

Source: https://www.replibyte.com/docs/datastores

Command to create a directory for local disk storage.

```bash
mkdir /data/replibyte
```

--------------------------------

### List existing database dumps

Source: https://www.replibyte.com/

Displays a table of all available database dumps with their metadata.

```bash
replibyte -c conf.yaml dump list  
  
type          name                  size    when                    compressed  encrypted  
PostgreSQL    dump-1647706359405    154MB   Yesterday at 03:00 am   true        true  
PostgreSQL    dump-1647731334517    152MB   2 days ago at 03:00 am  true        true  
PostgreSQL    dump-1647734369306    149MB   3 days ago at 03:00 am  true        true  
```

--------------------------------

### Run development seed command

Source: https://www.replibyte.com/docs/guides/deploy-replibyte/container

Restore the latest production dump into a development database.

```bash
docker run -e S3_ACCESS_KEY_ID=XXX \
           -e S3_SECRET_ACCESS_KEY=YYY \
           -e S3_REGION=us-east-2 \
           -e S3_BUCKET=my-test-bucket \
           -e SOURCE_CONNECTION_URI=postgres://... \
           -e DESTINATION_CONNECTION_URI=postgres://... \
           -e ENCRYPTION_SECRET=itIsASecret \
           ghcr.io/qovery/replibyte replibyte dump restore remote -v latest  
```

--------------------------------

### Run RepliByte with Docker

Source: https://www.replibyte.com/docs/getting-started/installation

Execute RepliByte using a Docker container with a mounted configuration file.

```bash
docker run -it --rm --name replibyte \
  -v "$(pwd)/replibyte.yaml:/replibyte.yaml:ro" \
  ghcr.io/qovery/replibyte --config replibyte.yaml  
```

--------------------------------

### List Available Transformers

Source: https://www.replibyte.com/docs/transformers

Command to display all pre-made transformers available in the Replibyte configuration.

```bash
replibyte -c conf.yaml transformer list  
   
 name            | description  
-----------------+--------------------------------------------------------------------------------------------  
 email           | Generate an email address (string only). [john.doe@company.com]->[tony.stark@avengers.com]  
 first-name      | Generate a first name (string only). [Lucas]->[Georges]  
 phone-number    | Generate a phone number (string only).  
 random          | Randomize value but keep the same length (string only). [AAA]->[BBB]  
 keep-first-char | Keep only the first character of the column.  
 transient       | Does not modify the value.  
 credit-card     | Generate a credit card number (string only).  
 redacted        | Obfuscate your sensitive data (string only). [4242 4242 4242 4242]->[424****************]  
 ...  
```

--------------------------------

### Basic Replibyte Configuration

Source: https://www.replibyte.com/docs/getting-started/configuration

Defines the core structure for connecting a source database to a destination via an S3-compatible datastore.

```yaml
encryption_key: $MY_PRIVATE_ENC_KEY # optional - encrypt data on datastore  
source:  
  connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL  
datastore:  
  aws:  
    bucket: $BUCKET_NAME  
    region: $S3_REGION  
    credentials:  
      access_key_id: $ACCESS_KEY_ID  
      secret_access_key: $AWS_SECRET_ACCESS_KEY  
destination:  
  connection_uri: postgres://user:password@host:port/db # you can use $DATABASE_URL  
```

--------------------------------

### Run RepliByte tests with environment variables

Source: https://www.replibyte.com/docs/contributing

Executes tests using specific AWS credentials for the MinIO datastore.

```bash
AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin cargo test
```

--------------------------------

### Replibyte configuration file

Source: https://www.replibyte.com/docs/guides/deploy-replibyte/container

Define the source database, data transformers, and S3 datastore settings in a file named replibyte.yaml.

```yaml
encryption_key: $ENCRYPTION_SECRET # put a secure secret here  
source:  
  connection_uri: $SOURCE_CONNECTION_URI  
  transformers:  
    - database: public  
      table: customers  
      columns:  
        - name: first_name  
          transformer_name: first-name  
        - name: last_name  
          transformer_name: random  
        - name: contact_phone  
          transformer_name: phone-number  
        - name: contact_email  
          transformer_name: email  
datastore:  
  aws:  
    bucket: $S3_BUCKET  
    region: $S3_REGION  
    access_key_id: $S3_ACCESS_KEY_ID  
    secret_access_key: $S3_SECRET_ACCESS_KEY  
destination:  
  connection_uri: $DESTINATION_CONNECTION_URI  
```

--------------------------------

### Configure database subsetting in conf.yaml

Source: https://www.replibyte.com/docs/guides/subset-a-dump

Add the `database_subset` object to your `conf.yaml` to enable subsetting. Specify the database, table, strategy, and options. Use `passthrough_tables` to keep entire tables.

```yaml
source:
  connection_uri: postgres://user:password@host:port/db
  transformers:
    - database: public
      table: customers
      columns:
        - name: first_name
          transformer_name: first-name
        - name: last_name
          transformer_name: random
        - name: contact_phone
          transformer_name: phone-number
        - name: contact_email
          transformer_name: email
database_subset:
  database: public
  table: customers
  strategy_name: random
  strategy_options:
    percent: 10
  passthrough_tables:
    - product_catalog

```

--------------------------------

### Run production dump command

Source: https://www.replibyte.com/docs/guides/deploy-replibyte/container

Execute a database dump operation in a production environment using environment variables.

```bash
docker run -e S3_ACCESS_KEY_ID=XXX \
           -e S3_SECRET_ACCESS_KEY=YYY \
           -e S3_REGION=us-east-2 \
           -e S3_BUCKET=my-test-bucket \
           -e SOURCE_CONNECTION_URI=postgres://... \
           -e DESTINATION_CONNECTION_URI=postgres://... \
           -e ENCRYPTION_SECRET=itIsASecret \
           ghcr.io/qovery/replibyte replibyte dump create  
```

--------------------------------

### Execute manual database dump commands

Source: https://www.replibyte.com/docs/guides/create-a-dump

Commands to manually generate database dumps for various database systems.

```bash
pg_dump --column-inserts --no-owner -h [host] -p [port] -U [username] [database]  

```

```bash
mysqldump -h [host] -P [port] -u [username] -p --add-drop-database --add-drop-table --skip-extended-insert --complete-insert --single-transaction --quick --databases  

```

```bash
mongodump -h [host] --port [port] --authenticationDatabase [auth_db|default: admin] --db [database] -u [username] -p [password] --archive  

```

--------------------------------

### Replibyte Configuration File

Source: https://www.replibyte.com/docs/guides/create-a-dump

Defines the source database connection, data transformers, and AWS datastore settings.

```yaml
source:  
  connection_uri: postgres://user:password@host:port/db # optional - use only for option #1  
  transformers:  
    - database: public  
      table: customers  
      columns:  
        - name: first_name  
          transformer_name: first-name  
        - name: last_name  
          transformer_name: random  
        - name: contact_phone  
          transformer_name: phone-number  
        - name: contact_email  
          transformer_name: email  
datastore:  
  aws:  
    bucket: my-replibyte-dumps  
    region: us-east-2  
    credentials:  
      access_key_id: $ACCESS_KEY_ID  
      secret_access_key: $AWS_SECRET_ACCESS_KEY  
      session_token: XXX # optional  
```

--------------------------------

### Configure source connection URI in conf.yaml

Source: https://www.replibyte.com/docs/guides/create-a-dump

Define the database connection URI in the configuration file for automated dumping.

```yaml
source:  
  connection_uri: postgres://[user]:[password]@[host]:[port]/[database]  

```

```yaml
source:  
  connection_uri: mysql://[user]:[password]@[host]:[port]/[database]  

```

```yaml
source:  
  connection_uri: mongodb://[user]:[password]@[host]:[port]/[database]  

```

```yaml
source:  
  connection_uri: $DATABASE_URL  

```

--------------------------------

### Configure datastore in conf.yaml

Source: https://www.replibyte.com/docs/guides/create-a-dump

Define the S3 bucket datastore settings for uploading transformed dumps.

```yaml
datastore:  
  aws:  
    bucket: my-replibyte-dumps  
    region: us-east-2  
    credentials:  
      access_key_id: $ACCESS_KEY_ID  
      secret_access_key: $AWS_SECRET_ACCESS_KEY  
      session_token: XXX # optional  

```

--------------------------------

### Environment variables file

Source: https://www.replibyte.com/docs/guides/deploy-replibyte/container

Define required environment variables in a text file for use with the Docker container.

```text
$ cat env.txt  
S3_ACCESS_KEY_ID  
S3_SECRET_ACCESS_KEY  
S3_REGION=us-east-2  
S3_BUCKET=my-test-bucket  
SOURCE_CONNECTION_URI=postgres://...  
DESTINATION_CONNECTION_URI=postgres://...  
ENCRYPTION_SECRET  
```

--------------------------------

### Configure Local Disk Datastore

Source: https://www.replibyte.com/docs/datastores

Configuration block for local disk storage. The directory must be readable and writable by the user running Replibyte.

```yaml
...
datastore:
  local_disk:
    dir: <your_directory_path>
...
```

```yaml
...
datastore:
  local_disk:
    dir: /data/replibyte
...
```

--------------------------------

### Configure S3-Compatible Datastore

Source: https://www.replibyte.com/docs/datastores

Configuration block for custom S3-compatible storage services requiring a specific endpoint.

```yaml
...
datastore:
  aws:
    bucket: <your_bucket>
    region: <your_region>
    credentials:
      access_key_id: XXX
      secret_access_key: XXX
    endpoint:
      custom: 'https://your-s3-compatible-endpoint'
...
```

--------------------------------

### List Available Dumps

Source: https://www.replibyte.com/docs/guides/delete-a-dump

This command lists all available dumps with their details, which is useful for identifying the name of the dump you want to delete.

```bash
replibyte -c conf.yaml dump list
```

--------------------------------

### Restore Dump to a File Locally

Source: https://www.replibyte.com/docs/guides/restore-a-dump

This command restores the latest dump and outputs its content to a specified file (e.g., dump.sql). This is useful for inspecting the dump's content manually. The -i parameter specifies the database type.

```bash
replibyte -c conf.yaml dump restore local -i postgres -v latest -o > dump.sql  

```

--------------------------------

### Restore Specific Dump Locally with Docker

Source: https://www.replibyte.com/docs/guides/restore-a-dump

Restore a particular dump by its name into a local Docker instance. Use the 'dump list' command to find the exact name of the dump you wish to restore.

```bash
replibyte -c conf.yaml dump restore local -d postgres -v dump-1647731334517  

```

--------------------------------

### SQL INSERT Statement with Masked Payment Card

Source: https://www.replibyte.com/docs/transformers

Demonstrates how a SQL INSERT statement is transformed to mask sensitive payment card information.

```sql
INSERT INTO public.my_table (payment_card) VALUE ('1234 1234 1234 1234');  

```

```sql
INSERT INTO public.my_table (payment_card) VALUE ('123####################');  

```

--------------------------------

### Build WASM Transformer

Source: https://www.replibyte.com/docs/advanced-guides/web-assembly-transformer

Compile your Rust project into a WASM binary for the wasm32-wasi target. The output will be located in the target directory.

```bash
cargo build --release --target wasm32-wasi  

```

--------------------------------

### Restore dump to local container

Source: https://www.replibyte.com/

Restores the latest dump into a local database container.

```bash
replibyte -c conf.yaml dump restore local -v latest -i postgres -p 5432  
```

--------------------------------

### Pull Replibyte Docker image

Source: https://www.replibyte.com/docs/guides/deploy-replibyte/container

Download the official Replibyte image from the GitHub Container Registry.

```bash
docker pull ghcr.io/qovery/replibyte  
```

--------------------------------

### Configure Replibyte YAML for WASM Transformer

Source: https://www.replibyte.com/docs/advanced-guides/web-assembly-transformer

Update your `replibyte.yaml` file to specify the path to your compiled WASM transformer. This tells Replibyte where to find and execute your custom logic.

```yaml
# ...  
   - database: <your-db-name>  
     table: <your-table-name>  
     columns:  
       - name: <your-column-name>  
         transformer_name: custom-wasm  
         transformer_options:  
           path: "path/to/your/custom-wasm-transformer.wasm"  
# ...  

```

--------------------------------

### Configure AWS S3 Datastore

Source: https://www.replibyte.com/docs/datastores

Configuration block for AWS S3 storage. Optional fields like profile, region, and explicit credentials can be omitted to use default AWS CLI mechanisms.

```yaml
...
datastore:
  aws:
    bucket: <your_bucket>
    profile: <your_profile> # optional
    region: <your_region> # optional
    credentials: # optional
      access_key_id: XXX
      secret_access_key: XXX
      session_token: XXX # optional
...
```

--------------------------------

### Configure GCP Cloud Storage Datastore

Source: https://www.replibyte.com/docs/datastores

Configuration block for GCP Cloud Storage using S3-compatible interoperability keys.

```yaml
...
datastore:
  gcp:
    bucket: your-bucket-name
    region: us-central1
    access_key: $GS_ACCESS_KEY
    secret: $GS_SECRET
...
```

--------------------------------

### Restore dump to remote database

Source: https://www.replibyte.com/

Restores the latest dump into a remote database instance.

```bash
replibyte -c conf.yaml dump restore remote -v latest  
```

--------------------------------

### Configure transformers in conf.yaml

Source: https://www.replibyte.com/docs/guides/create-a-dump

Apply data masking transformers to specific table columns in the configuration file.

```yaml
source:  
  connection_uri: postgres://user:password@host:port/db  
  transformers:  
    - database: public  
      table: customers  
      columns:  
        - name: first_name  
          transformer_name: first-name  
        - name: last_name  
          transformer_name: random  
        - name: contact_phone  
          transformer_name: phone-number  
        - name: contact_email  
          transformer_name: email  

```

--------------------------------

### Rust WASM Transformer Implementation

Source: https://www.replibyte.com/docs/advanced-guides/web-assembly-transformer

This Rust code reads a string from stdin, reverses it, and prints the result to stdout. It serves as the core logic for the WASM transformer.

```rust
// This is actually the source of the `.wasm` file in this example. Feel free to edit it !  
fn main() {  
    // Read input value from stdin  
    let mut input = String::new();  
    std::io::stdin().read_line(&mut input).unwrap();  
      
    // Transform the value as you see fit (in this case we just reverse the string)  
    let output: String = input.chars().rev().collect();  
      
    // Write transformed value to stdout (simply print)  
    println!("{}", output);
}

```

--------------------------------

### Add WASM Target for Rust

Source: https://www.replibyte.com/docs/advanced-guides/web-assembly-transformer

Add the wasm32-wasi target to your Rust toolchain to enable cross-compilation for WebAssembly.

```bash
rustup target add wasm32-wasi  

```

--------------------------------

### Restore Latest Dump Remotely

Source: https://www.replibyte.com/docs/guides/restore-a-dump

Restore the latest dump to a remote database. Ensure the 'destination.connection_uri' is correctly configured in your conf.yaml file. The 'wipe_database' option can be set to false to prevent wiping the destination database.

```bash
replibyte -c conf.yaml dump restore remote -v latest  

```

--------------------------------

### Delete a Dump by Name

Source: https://www.replibyte.com/docs/guides/delete-a-dump

Use this command to delete a specific dump file by its name. You can list available dumps using `replibyte -c conf.yaml dump list`.

```bash
replibyte -c conf.yaml dump delete <DUMP_NAME>
```

--------------------------------

### Delete Dumps Older Than a Specified Number of Days

Source: https://www.replibyte.com/docs/guides/delete-a-dump

This command deletes all dumps that are older than the specified number of days. Currently, only the day unit ('d') is supported.

```bash
replibyte -c conf.yaml dump delete --older-than=2d
```

--------------------------------

### Keep only a maximum number of dumps

Source: https://www.replibyte.com/docs/guides/delete-a-dump

Deletes older dumps to ensure only a specified maximum number of recent dumps are retained.

```APIDOC
## DELETE dump delete --keep-last

### Description
Retains only the most recent N dumps and deletes the rest.

### Method
DELETE

### Endpoint
dump delete --keep-last=<number>

### Parameters
#### Query Parameters
- **keep-last** (integer) - Required - The maximum number of dumps to keep.
```

--------------------------------

### Delete a dump by name

Source: https://www.replibyte.com/docs/guides/delete-a-dump

Deletes a specific database dump identified by its unique name.

```APIDOC
## DELETE dump delete <DUMP_NAME>

### Description
Deletes a specific dump by its name.

### Method
DELETE

### Endpoint
dump delete <DUMP_NAME>

### Parameters
#### Path Parameters
- **DUMP_NAME** (string) - Required - The unique name of the dump to delete.
```

--------------------------------

### Delete dumps by age

Source: https://www.replibyte.com/docs/guides/delete-a-dump

Deletes all database dumps that are older than a specified number of days.

```APIDOC
## DELETE dump delete --older-than

### Description
Deletes dumps older than a specified number of days.

### Method
DELETE

### Endpoint
dump delete --older-than=<days>d

### Parameters
#### Query Parameters
- **older-than** (string) - Required - The age threshold in days (e.g., 2d).
```

--------------------------------

### Keep Only the Last N Dumps

Source: https://www.replibyte.com/docs/guides/delete-a-dump

This command deletes older dumps, ensuring that only the specified number of the most recent dumps are kept.

```bash
replibyte -c conf.yaml dump delete --keep-last=10
```

--------------------------------

### Define AWS S3 IAM Policy

Source: https://www.replibyte.com/docs/datastores

IAM policy granting read and write access to a specific S3 bucket.

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name-here",
                "arn:aws:s3:::your-bucket-name-here/*"
            ]
        }
    ]
}
```

--------------------------------

### MongoDB Embedded Array of Documents Transformation Configuration

Source: https://www.replibyte.com/docs/transformers

Configuration for transforming email and phone number fields within an array of embedded documents in MongoDB.

```yaml
source:
  connection_uri: $DATABASE_URL
  transformers:
    - database: my_database
      table: my_collection
      columns:
        - name: contacts.$[].email
          transformer_name: email
        - name: contacts.$[].phone_number
          transformer_name: phone-number

```

--------------------------------

### MongoDB Embedded Sub-document Transformation Configuration

Source: https://www.replibyte.com/docs/transformers

Configuration for transforming email and phone number fields within an embedded sub-document in MongoDB.

```yaml
source:
  connection_uri: $DATABASE_URL
  transformers:
    - database: my_database
      table: my_collection
      columns:
        - name: contact.email
          transformer_name: email
        - name: contact.phone_number
          transformer_name: phone-number

```
Tokens:
Raw