### GET /debug/pprof/profile Source: https://github.com/alibaba/mongoshake/wiki/FAQ Fetches a CPU profile for debugging. ```APIDOC ## GET /debug/pprof/profile ### Description Fetches a CPU profile for debugging. ### Method GET ### Endpoint /debug/pprof/profile ### Parameters None ### Response #### Success Response (200) - **body** (binary) - CPU profile data #### Response Example (binary data) ``` -------------------------------- ### Download govendor dependency manager Source: https://github.com/alibaba/mongoshake/wiki/tutorial-about-how-to-set-up Install govendor, required to download MongoShake's dependencies before compiling. ```bash go get -u github.com/kardianos/govendor ``` -------------------------------- ### GET /debug/pprof/heap Source: https://github.com/alibaba/mongoshake/wiki/FAQ Fetches heap profile for debugging. ```APIDOC ## GET /debug/pprof/heap ### Description Fetches heap profile for debugging. ### Method GET ### Endpoint /debug/pprof/heap ### Parameters None ### Response #### Success Response (200) - **body** (binary) - Heap profile data #### Response Example (binary data) ``` -------------------------------- ### GET /debug/pprof/goroutine?debug=2 Source: https://github.com/alibaba/mongoshake/wiki/FAQ Fetches the goroutine status for debugging purposes. ```APIDOC ## GET /debug/pprof/goroutine?debug=2 ### Description Fetches the goroutine status for debugging purposes. ### Method GET ### Endpoint /debug/pprof/goroutine?debug=2 ### Parameters #### Query Parameters - **debug** (integer) - Optional - Debug level (2 for full goroutine dump) ### Response #### Success Response (200) - **body** (string) - Goroutine dump output #### Response Example (goroutine dump text) ``` -------------------------------- ### Checkpoint update warning log when lacking write permission Source: https://github.com/alibaba/mongoshake/wiki/FAQ Example warning log indicating that the checkpoint update failed due to lack of write permission on the 'mongoshake' database. This log appears when the user does not have write permission for the checkpoint collection. ```text [WARN] [collector.(*OplogSyncer).checkpoint:64] CheckpointOperation updated is not suitable. lowest [6769220508375318558]. current [6769212060174647296]. reason : not authorized on mongoshake to execute command { update: "ckpt_default_singapore_12_12", writeConcern: { getLastError: 1 }, ordered: true, $db: "mongoshake" } ``` -------------------------------- ### Build MongoShake from source Source: https://github.com/alibaba/mongoshake/blob/develop/README.md Build MongoShake from source. Requires Go version >= 15.10. Clone the repository, change to the MongoShake directory, run make, and then run the collector with the configuration file. ```bash git clone https://github.com/alibaba/MongoShake.git ``` ```bash cd MongoShake ``` ```bash make ``` ```bash ./bin/collector -conf=conf/collector.conf ``` -------------------------------- ### Run MongoShake collector with configuration Source: https://github.com/alibaba/mongoshake/wiki/tutorial-about-how-to-set-up Execute the collector binary in the bin directory, specifying the configuration file. Add -verbose to print logs to stdout. ```bash ./collector -conf=collector.conf ``` -------------------------------- ### Run pre-built collector binary Source: https://github.com/alibaba/mongoshake/blob/develop/README.md Run the pre-built collector binary for OSX or Linux. Choose the appropriate binary for your platform. ```bash ./bin/collector.darwin ``` ```bash collector.linux ``` -------------------------------- ### Run the sharded orphan filter integration test Source: https://github.com/alibaba/mongoshake/blob/develop/tests/integration/sharded/README.md Commands to set up the cluster, inject an orphan, verify the filter, and tear down. Requires Docker, pymongo, and a built collector binary in ../../../bin/. ```bash cd tests/integration/sharded docker compose up -d ./setup_cluster.sh # init replsets + addShard + stopBalancer ./inject_orphan.py # craft orphan scenario ./verify_orphan_filter.py ../../../bin/collector.darwin # or collector.linux docker compose down -v # tear down ``` -------------------------------- ### ExampleReplayer.handler() function in receiver/replayer.go Source: https://github.com/alibaba/mongoshake/wiki/FAQ This function processes messages from the pendingQueue, parses batched oplogs, executes a callback if present, and updates the ack timestamp. Users can add custom logic after the comment 'add logical code below'. ```go func (er *ExampleReplayer) handler() { for msg := range er.pendingQueue { count := uint64(len(msg.message.RawLogs)) if count == 0 { // may be probe request continue } // parse batched message oplogs := make([]*oplog.PartialLog, len(msg.message.RawLogs), len(msg.message.RawLogs)) for i, raw := range msg.message.RawLogs { oplogs[i] = &oplog.PartialLog{} bson.Unmarshal(raw, &oplogs[i]) oplogs[i].RawSize = len(raw) LOG.Info(oplogs[i]) // just print for test } if callback := msg.completion; callback != nil { callback() // exec callback if exist } // get the newest timestamp n := len(oplogs) lastTs := utils.TimestampToInt64(oplogs[n - 1].Timestamp) er.Ack = lastTs // add logical code below } } ``` -------------------------------- ### Run integration tests for lazy oplog parsing Source: https://github.com/alibaba/mongoshake/blob/develop/docs/lazy-oplog-bson.md Command to run integration tests that require a live MongoDB replica set and sharded cluster. Set `MONGOSHAKE_TEST_URL` and `MONGOSHAKE_TEST_URL_SHARDING` before running. ```sh GOWORK=off go test -race ./oplog ./executor ``` -------------------------------- ### Run integration test for transaction checkpoint pin Source: https://github.com/alibaba/mongoshake/blob/develop/docs/fix-txn-checkpoint-pin.md Command to run the integration test that reproduces the loss window end-to-end. Requires the integration build tag and a 10-minute timeout. ```bash go test -tags integration ./integration -run TestTxnCheckpointPin -v -timeout 10m ``` -------------------------------- ### Run network-free tests for lazy oplog parsing Source: https://github.com/alibaba/mongoshake/blob/develop/docs/lazy-oplog-bson.md Commands to run the network-free test suite covering the oplog, filter, transform, executor, and collector packages. The `TestConvertEvent2Oplog` integration test is excluded because it requires a live MongoDB. ```sh GOWORK=off go test -race ./oplog -skip '^TestConvertEvent2Oplog$' GOWORK=off go test -race ./collector/filter ./collector/transform GOWORK=off go test -race ./executor -run 'Test(Lazy|TransformLog|MergeToGroups|GetFieldValue|SplitDotted|ResolveConflictFilter|ParseDupKey)' GOWORK=off go test ./collector -run '^TestDeserializer$' GOWORK=off go test ./oplog -run '^$' -bench '^BenchmarkParseRaw$' -benchmem -count=3 ``` -------------------------------- ### Default checkpoint configuration in collector.conf Source: https://github.com/alibaba/mongoshake/wiki/FAQ Default checkpoint configuration variables: context.storage, context.address, and context.start_position. context.storage can be 'database' or 'api'; context.address is the collection name in the 'mongoshake' database; context.start_position is used only when no checkpoint exists. ```ini context.storage = database context.address = ckpt_default context.start_position = 2000-01-01T00:00:01Z ``` -------------------------------- ### Debug MongoShake with pprof commands Source: https://github.com/alibaba/mongoshake/wiki/FAQ Use these commands to debug MongoShake via its pprof port (default 9200). Fetch goroutine status with curl, and use go tool pprof to collect CPU profile and heap status. ```bash curl http://127.0.0.1:9200/debug/pprof/goroutine?debug=2 # use curl to fetch the routine status go tool pprof http://127.0.0.1:9200/debug/pprof/profile # use go tool command to fetch the profile go tool pprof -top http://127.0.0.1:9200/debug/pprof/heap # use go tool command to fetch the heap status ``` -------------------------------- ### Run mongoshake-stat to monitor MongoShake via RESTful API Source: https://github.com/alibaba/mongoshake/wiki/FAQ Use the mongoshake-stat script to monitor MongoShake in real time. The output is refreshed every second and shows metrics like logs_get/sec, logs_repl/sec, logs_success/sec, and timestamps. The script connects to the RESTful API on the specified port (default 9100). ```bash vinllen@ ~/code/mongo-shake-github/mongo-shake/bin$ ./mongoshake-stat --port=9100 |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| | logs_get/sec | logs_repl/sec | logs_success/sec | lsn.time | lsn_ack.time | lsn_ckpt.time | now.time | replset | |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| | none | none | none | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-31 11:55:03 |zz-mgset-source-20180711 | |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| | 0 | 0 | 0 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-31 11:55:04 |zz-mgset-source-20180711 | |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| | 1 | 0 | 0 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-31 11:55:05 |zz-mgset-source-20180711 | |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| | 0 | 0 | 0 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-31 11:55:06 |zz-mgset-source-20180711 | |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| | 0 | 0 | 0 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-31 11:55:07 |zz-mgset-source-20180711 | |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| | 0 | 0 | 0 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-29 23:10:41 | 2018-07-31 11:55:08 |zz-mgset-source-20180711 | |---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------| ``` -------------------------------- ### Check current TPS via RESTful API Source: https://github.com/alibaba/mongoshake/wiki/FAQ Use this curl command to check the current TPS setting from the sentinel endpoint. ```bash curl 127.0.0.1:9100/sentinel ``` -------------------------------- ### Pause the synchronization link via RESTful API Source: https://github.com/alibaba/mongoshake/wiki/FAQ Use this curl command to pause the MongoShake synchronization link by setting the Pause option to true. ```bash curl -X POST --data '{"Pause": true}' 127.0.0.1:9100/sentinel/options ``` -------------------------------- ### Set TPS to 1000 via RESTful API Source: https://github.com/alibaba/mongoshake/wiki/FAQ Use this curl command to limit the MongoShake pulling speed (TPS) to 1000 by posting to the sentinel options endpoint. ```bash curl -X POST --data '{"TPS": 1000}' 127.0.0.1:9100/sentinel/options ```