MongoDB Mongosync

https://www.mongodb.com/docs/mongosync/
Admin
MongoDB Mongosync is a data migration tool that replicates data and writes from one MongoDB cluster...
Tokens:83,400
Snippets:730
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Start Multiple mongosync Instances

Source: https://www.mongodb.com/docs/mongosync/current/topologies/multiple-mongosyncs

Use the start command to initiate synchronization on each mongosync instance. The start command options must be identical for all instances.

```shell
curl mongosync01Host:27601/api/v1/start -XPOST --data \
     '{ "source": "cluster0", "destination": "cluster1", \
        "reversible": false }'

curl mongosync02Host:27602/api/v1/start -XPOST --data \
     '{ "source": "cluster0", "destination": "cluster1", \
        "reversible": false }'
```

--------------------------------

### Example log file listing

Source: https://www.mongodb.com/docs/mongosync/current/reference/logging

Example output showing the active log file and rotated historical logs.

```text
mongosync-2022-06-17T16-27-58.187.log  mongosync.log
mongosync-2022-06-21T13-31-42.668.log
```

--------------------------------

### Start mongosync with Filter via cURL

Source: https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering

Example of using the /start API endpoint with a cURL command to initiate a sync with specific namespaces included.

```bash
curl -X POST "http://localhost:27182/api/v1/start" --data '
   {
      "source": "cluster0",
      "destination": "cluster1",
      "includeNamespaces": [
         {
             "database": "sales",
             "collectionsRegex": {
                "pattern": "^accounts_.+$",
                "options": "i"
             }
         }, {
            "database": "marketing"
         }
      ]
   } '
```

--------------------------------

### Start migration-verifier process

Source: https://www.mongodb.com/docs/mongosync/current/reference/verification/verifier

Initiate the migration-verifier process by specifying the source, destination, and metadata URIs.

```bash
migration-verifier --verifyAll \
   --srcURI example.net:27020 \
   --destURI example.net:27021 \
   --metaURI example.net:27017
```

--------------------------------

### POST /api/v1/start

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/start

Starts the synchronization between a source and destination cluster.

```APIDOC
## POST /api/v1/start

### Description
Starts the synchronization between a source and destination cluster. The mongosync instance must be in the IDLE state to execute this command.

### Method
POST

### Endpoint
/api/v1/start

### Parameters
#### Request Body
- **source** (string) - Required - Name of the source cluster.
- **destination** (string) - Required - Name of the destination cluster.
```

--------------------------------

### Regular Expression Options Example

Source: https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering/filter-regex

An example of an includeNamespaces filter using a regex pattern with 'ms' options to match collections starting with 'accounts_' in the 'sales' database.

```json
"includeNamespaces": [
   {
      "database": "sales",
      "collectionsRegex": {
         "pattern": "^accounts_.+?$",
         "options": "ms"
      }
   }
]
```

--------------------------------

### Start mongosync with Configuration File

Source: https://www.mongodb.com/docs/mongosync/current/reference/configuration

Command to launch the mongosync process using a specified configuration file path.

```shell
mongosync --config <configPath>
```

--------------------------------

### Start mongosync with configuration

Source: https://www.mongodb.com/docs/mongosync/current/reference/logging

Default execution of mongosync using a configuration file, which outputs logs to stdout.

```shell
$ mongosync --config /etc/mongosync.conf
```

--------------------------------

### Example Atlas Connection Strings

Source: https://www.mongodb.com/docs/mongosync/current/connecting/atlas-to-atlas

Example connection strings for two separate Atlas clusters, cluster0 and cluster1.

```shell
cluster0:
mongodb+srv://clusterAdmin:superSecret@cluster1Name.abc123.mongodb.net
cluster1:
mongodb+srv://clusterAdmin:superSecret@cluster2Name.abc123.mongodb.net
```

--------------------------------

### Start API Request Body Parameters

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/start

Details the configuration parameters for initiating a migration sync operation.

```APIDOC
### Request Body Parameters

- **preExistingDestinationData** (boolean) - Optional - Allows pre-existing namespaces on the destination cluster. Default is false.
- **reversible** (boolean) - Optional - Enables the sync operation to be reversed. Default is false.
- **sharding** (document) - Optional - Configures sync between a replica set and sharded cluster.
- **verification** (document) - Optional - Configures the embedded verifier.
- **verification.enabled** (boolean) - Optional - Enables the embedded verifier. Enabled by default.
```

--------------------------------

### Example formatted log output

Source: https://www.mongodb.com/docs/mongosync/current/reference/logging

Sample output of log entries after being processed by jq.

```json
{
   "level": "info",
   "mongosyncID": "shard02",
   "verbosity": "INFO",
   "id": "shard02",
   "port":27301,
   "time": "2022-06-21T11:15:33-04:00",
   "message": "Mongosync Options"
}
{
   "level": "info",
   "mongosyncID": "shard02",
   "time": "2022-06-21T11:15:33-04:00",
   "message": "Initialized client0 with URI: mongodb://192.0.2.1:27130 and client1 with URI: mongodb://192.0.2.2:27140."
}
{
   "level": "info",
   "mongosyncID": "shard02",
   "time": "2022-06-21T11:15:33-04:00",
   "message": "Preflight checks completed."
}
{
   "level": "info",
   "mongosyncID": "shard02",
   "time": "2022-06-21T11:15:33-04:00",
   "message": "Launch replication thread"
}
```

--------------------------------

### Start verification checks

Source: https://www.mongodb.com/docs/mongosync/current/reference/verification/verifier

Send a POST request to the Migration Verifier API to begin the verification process.

```bash
curl -H "Content-Type: application/json" \
    -X POST -d '{}' http://127.0.0.1:27020/api/v1/check
```

--------------------------------

### Start a Sync Job

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/start

Initiates a standard sync job between source cluster cluster0 and destination cluster cluster1.

```shell
curl localhost:27182/api/v1/start -XPOST \
--data '
   {
      "source": "cluster0",
      "destination": "cluster1"
   } '
```

```json
{"success":true}
```

--------------------------------

### Connection String for cluster0

Source: https://www.mongodb.com/docs/mongosync/current/connecting/onprem-to-atlas

Example connection string configured for the self-managed cluster0 using the gathered connection information.

```text
cluster0:
mongodb://clusterAdmin:superSecret@clusterOne01.fancyCorp.com:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020
```

--------------------------------

### Get synchronization progress

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/progress

Request the current status of the synchronization process using a GET request.

```shell
curl localhost:27182/api/v1/progress -XGET
```

--------------------------------

### POST /start

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/start

Initiates the synchronization process between the source and destination clusters.

```APIDOC
## POST /start

### Description
Initiates the synchronization process between the source and destination clusters.

### Method
POST

### Endpoint
/start

### Request Body
- **buildIndexes** (string) - Optional - Configures index builds during sync. Supported options: "afterDataCopy", "beforeDataCopy", "excludeHashed", "excludeHashedAfterCopy", "never".
```

--------------------------------

### Reverse Sync Operation Example

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/reverse

Example of using curl to reverse the sync direction and the corresponding JSON response.

```shell
curl localhost:27182/api/v1/reverse -XPOST --data '{ }'
```

```json
{"success":true}
```

--------------------------------

### Example Connection Strings for Clusters

Source: https://www.mongodb.com/docs/mongosync/current/connecting/onprem-to-onprem

Specific connection strings constructed for cluster0 and cluster1 using the provided administrative credentials and server addresses.

```shell
cluster0:
mongodb://clusterAdmin:superSecret@clusterOne01.fancyCorp.com:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020
cluster1:
mongodb://clusterAdmin:superSecret@clusterTwo01.fancyCorp.com:20020,clusterTwo02.fancyCorp.com:20020,clusterTwo03.fancyCorp.com:20020
```

--------------------------------

### Start migration with all databases and collections in natural order

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/start

Use the copyInNaturalOrder option with a dot (.) to copy all databases and their collections in their natural order.

```shell
curl -X POST "http://localhost:27182/api/v1/start" --data '
  {
    "source": "cluster0",
    "destination": "cluster1",
    "copyInNaturalOrder": [
      {
        "database": ".",
      },
    ]
  }'
```

```json
{"success":true}
```

--------------------------------

### Pause Synchronization Example

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/pause

This example demonstrates how to pause the synchronization operation using curl, including the required empty object in the data payload, followed by the successful JSON response.

```shell
curl localhost:27182/api/v1/pause -XPOST --data '{ }'
```

```json
{"success":true}
```

--------------------------------

### GET /api/v1/progress

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/progress

Retrieves the current status of the synchronization process.

```APIDOC
## GET /api/v1/progress

### Description
Returns the current status of the synchronization process or an error.

### Method
GET

### Endpoint
/api/v1/progress

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the command succeeded.
- **progress** (object) - Contains the synchronization status details.
  - **state** (string) - The current state of mongosync.
  - **canCommit** (boolean) - Indicates if a commit request will succeed.
  - **canWrite** (boolean) - Indicates if writes are permitted on the destination cluster.
  - **indexBuilding** (object) - Real-time progress of index builds.
    - **indexesBuilt** (integer) - Number of indexes finished building.
    - **totalIndexesToBuild** (integer) - Total number of indexes to build.
    - **collectionsFinished** (integer) - Number of collections with completed index builds.
    - **collectionsTotal** (integer) - Total number of collections requiring index builds.
  - **info** (string) - Extra information on synchronization progress.
  - **lag** (object) - Synchronization lag metrics.
    - **overallLagSeconds** (integer) - Time difference in seconds between source and destination.
    - **crudLagSeconds** (integer) - CRUD component of lag in seconds.
    - **ddlLagSeconds** (integer) - DDL component of lag in seconds.
```

--------------------------------

### Example mongosync YAML Configuration File

Source: https://www.mongodb.com/docs/mongosync/current/reference/configuration

A sample YAML configuration file defining cluster connection strings, log path, and verbosity level.

```yaml
cluster0: "mongodb://192.0.2.10:27017"
cluster1: "mongodb://192.0.2.20:27017"
logPath: "/var/log/mongosync"
verbosity: "WARN"
```

--------------------------------

### GET /api/v1/progress

Source: https://www.mongodb.com/docs/mongosync/current/reference/verification/verifier

Retrieves the current progress and status of the verification tasks.

```APIDOC
## GET /api/v1/progress

### Description
Returns the current status of the verification process, including task counts and the current phase.

### Method
GET

### Endpoint
http://127.0.0.1:27020/api/v1/progress

### Response
#### Success Response (200)
- **progress** (object) - Contains the current phase, error status, and verification task metrics.

#### Response Example
{
   "progress": {
      "phase": "idle",
      "error": null,
      "verificationStatus": {
         "totalTasks": 1,
         "addedTasks": 0,
         "processingTasks": 0,
         "failedTasks": 0,
         "completedTasks": 0,
         "metadataMismatchTasks": 0,
         "recheckTasks": 0
      }
   }
}
```

--------------------------------

### GET /progress

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/progress

Retrieves the current progress and state of the mongosync operation.

```APIDOC
## GET /progress

### Description
Returns the current status, state, and progress metrics of the mongosync process. This endpoint does not auto-refresh and must be called repeatedly to get updated status.

### Method
GET

### Endpoint
/progress

### Response
#### Success Response (200)
- **state** (string) - The current state of the mongosync process (e.g., IDLE, PAUSED, INITIALIZING).
- **canCommit** (boolean) - Indicates if the process is ready to commit.
- **estimatedCopiedBytes** (number) - Best-effort estimate of bytes copied during collection copy.
- **estimatedTotalBytes** (number) - Best-effort estimate of total bytes to be copied.
- **lag** (object) - Current lag information (null if PAUSED).
- **warnings** (array) - List of warnings if user writes are blocked.

#### Error Response
- **success** (boolean) - Status of the command (false if failed).
- **error** (string) - Type of error.
- **errorDescription** (string) - Detailed description of the error.
```

--------------------------------

### Start with the Verifier Disabled

Source: https://www.mongodb.com/docs/mongosync/current/reference/api/start

Disables the embedded verifier by setting verification.enabled to false.

```text
curl localhost:27182/api/v1/start -XPOST \
--data '
   {
      "source": "cluster0",
      "destination": "cluster1",
      "verification": {
         "enabled": false
      }
   } '
```

```json
{"success":true}
```

--------------------------------

### Run First mongosync Instance

Source: https://www.mongodb.com/docs/mongosync/current/topologies/multiple-mongosyncs

Start the first mongosync instance for shard01 using the specified cluster connection strings and port.

```bash
mongosync \
  --cluster0 "mongodb://user:password@cluster0host:27500" \
  --cluster1 "mongodb://user:password@cluster1host:27500" \
  --id shard01 --port 27601
```

--------------------------------

### Start Sync API Request and Response

Source: https://www.mongodb.com/docs/mongosync/current/reference/verification/embedded

Use the /start endpoint to initiate data synchronization between clusters, followed by the success response.

```shell
curl localhost:27182/api/v1/start -XPOST \
    --data ' {
        "source": "cluster0",
        "destination": "cluster1",
    } '
```

```json
{"success":true}
```
Tokens:
Raw