{"id":15029559,"url":"https://github.com/alibaba/mongoshake","last_synced_at":"2026-04-28T09:06:45.690Z","repository":{"id":37706365,"uuid":"138833025","full_name":"alibaba/MongoShake","owner":"alibaba","description":"MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具，可以满足迁移和同步的需求，进一步实现灾备和多活功能。","archived":false,"fork":false,"pushed_at":"2026-04-21T09:44:07.000Z","size":81130,"stargazers_count":1807,"open_issues_count":22,"forks_count":460,"subscribers_count":43,"default_branch":"develop","last_synced_at":"2026-04-21T11:37:00.577Z","etag":null,"topics":["active-active","mongo-database","mongodb-oplog","mongoshake","replication"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alibaba.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-06-27T05:18:22.000Z","updated_at":"2026-04-21T09:56:23.000Z","dependencies_parsed_at":"2024-06-18T18:42:09.867Z","dependency_job_id":"0821b929-7678-4327-b5f0-bde6cb002da8","html_url":"https://github.com/alibaba/MongoShake","commit_stats":{"total_commits":572,"total_committers":21,"mean_commits":"27.238095238095237","dds":0.3951048951048951,"last_synced_commit":"acbaf609146b0b5e4c3f1c0589659468976a68cb"},"previous_names":["aliyun/mongo-shake"],"tags_count":69,"template":false,"template_full_name":null,"purl":"pkg:github/alibaba/MongoShake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FMongoShake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FMongoShake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FMongoShake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FMongoShake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alibaba","download_url":"https://codeload.github.com/alibaba/MongoShake/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FMongoShake/sbom","scorecard":{"id":183628,"data":{"date":"2025-08-11","repo":{"name":"github.com/alibaba/MongoShake","commit":"b05864e209f3901fc026189452e0a5c0214d7bdf"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.9,"checks":[{"name":"Code-Review","score":0,"reason":"Found 2/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":10,"reason":"0 commit(s) and 26 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":9,"reason":"binaries present in source code","details":["Warn: binary detected: tools/govendor:1"],"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: GNU General Public License v3.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact release-v2.8.5-20250403 not signed: https://api.github.com/repos/alibaba/MongoShake/releases/210054526","Warn: release artifact release-v2.8.4-20230425 not signed: https://api.github.com/repos/alibaba/MongoShake/releases/100781628","Warn: release artifact release-v2.8.3-20230329 not signed: https://api.github.com/repos/alibaba/MongoShake/releases/97420200","Warn: release artifact release-v2.8.2-20221215 not signed: https://api.github.com/repos/alibaba/MongoShake/releases/86177946","Warn: release artifact release-v2.8.1-20220922 not signed: https://api.github.com/repos/alibaba/MongoShake/releases/77837674","Warn: release artifact release-v2.8.5-20250403 does not have provenance: https://api.github.com/repos/alibaba/MongoShake/releases/210054526","Warn: release artifact release-v2.8.4-20230425 does not have provenance: https://api.github.com/repos/alibaba/MongoShake/releases/100781628","Warn: release artifact release-v2.8.3-20230329 does not have provenance: https://api.github.com/repos/alibaba/MongoShake/releases/97420200","Warn: release artifact release-v2.8.2-20221215 does not have provenance: https://api.github.com/repos/alibaba/MongoShake/releases/86177946","Warn: release artifact release-v2.8.1-20220922 does not have provenance: https://api.github.com/repos/alibaba/MongoShake/releases/77837674"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1","Warn: containerImage not pinned by hash: Dockerfile:7: pin your Docker image by updating alpine:3.12 to alpine:3.12@sha256:c75ac27b49326926b803b9ed43bf088bc220d22556de1bc5f72d742c91398f69","Info:   0 out of   2 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 3 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-16T19:21:30.983Z","repository_id":37706365,"created_at":"2025-08-16T19:21:30.983Z","updated_at":"2025-08-16T19:21:30.983Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32373566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T20:07:02.737Z","status":"online","status_checked_at":"2026-04-28T02:00:07.250Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["active-active","mongo-database","mongodb-oplog","mongoshake","replication"],"created_at":"2024-09-24T20:11:01.614Z","updated_at":"2026-04-28T09:06:45.678Z","avatar_url":"https://github.com/alibaba.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is a brief introduction of Mongo-Shake, please visit [english wiki](https://github.com/aliyun/mongo-shake/wiki/MongoShake-Detailed-Documentation) or [chinese wiki](https://yq.aliyun.com/articles/603329) if you want to see more details including architecture, data flow, performance test, business showcase and so on.\n\n*  [English document](https://github.com/aliyun/mongo-shake/wiki/MongoShake-Detailed-Documentation)\n*  [中文架构介绍文档](https://yq.aliyun.com/articles/603329)\n*  [第一次使用，如何配置](https://github.com/alibaba/MongoShake/wiki/%E7%AC%AC%E4%B8%80%E6%AC%A1%E4%BD%BF%E7%94%A8%EF%BC%8C%E5%A6%82%E4%BD%95%E8%BF%9B%E8%A1%8C%E9%85%8D%E7%BD%AE%EF%BC%9F)\n*  [Tutorial](https://github.com/alibaba/MongoShake/wiki/tutorial-about-how-to-set-up)\n*  [FAQ document](https://github.com/aliyun/mongo-shake/wiki/FAQ)\n*  [MongoShake最佳实践](https://yq.aliyun.com/articles/719704)\n*  [Performance test document](https://github.com/aliyun/mongo-shake/wiki/MongoShake-Performance-Document)\n\n# Mongo-Shake\n---\nMongo-Shake is developed and maintained by Nosql Team in Alibaba-Cloud.\u003cbr\u003e\nMongo-Shake is a universal platform for services based on MongoDB's oplog. It fetches oplog from source mongo database, and replays in the target mongo database or sends to other ends in different tunnels. If the target side is mongo database which means replay oplog directly, it's like a syncing tool that used to copy data from source MongoDB to another MongoDB to build redundant replication or active-active replication. Except for this direct way, there are others tunnel types such like rpc, file, tcp, kafka. Receivers wrote by users must define their own interfaces to connecting to these tunnels respectively. Users can also define their own tunnel type which is pluggable. If connecting to a third-party message middleware like kafka, the consumer can get the subscriber data in an asynchronous way in pub/sub module flexibly.\nHere comes general data flow, \u003cbr\u003e\n![pic1](resources/dataflow.png)\u003cbr\u003e\nThe source can be either single mongod, replica set or sharding while target can be mongod or mongos. If the source is replica set, we suggest fetching data from secondary/hidden to ease the primary pressure. If the source is sharding, every shard should connect to Mongo-Shake. There can be several mongos on the target side to keep high availability, and different data will be hashed and written to different mongos.\n\n# Parallel Replication\n---\nThere are three options for parallel replication which we call 'shad\\_key': __id__, __collection__ and __auto__. __id__ means the concurrency granularity is document while __collection__ means the granularity is collection/table. __auto__ option is decided by if there has unique index of any collection, it will change to __collection__ with unique index exist otherwise __id__.\n\n# High Availability\n---\nMongo-Shake periodically persistent its context into register center which by default is the source database. Currently, the context is checkpoint which marks the position of successfully replay oplog.\u003cbr\u003e\nHypervisor mechanism is also supported so that it will restart immediately when dies(`master_quorum` in configuration).\n`master_quorum` only supports `checkpoint.storage=database`; when `master_quorum=true`, `master_quorum.election_id` must be set to a valid MongoDB ObjectID and kept unique per independent sync job or HA group.\n\n# Filter\n---\nSupport filtering database and collection namespace with whitelist and blacklist.\n\n# DDL Syncing\n---\nStarting with version 1.5, MongoShake supports syncing DDL by using global barrier. Once fetching DDL oplog, MongoShake adds a barrier so that all the subsequent oplogs wait in the queue until this oplog is written into the target MongoDB or tunnel and the checkpoint is updated. Currently, DDL is only support for ReplicaSet on the source side(target side can be RelicaSet or Sharding), we will support Sharding in the later version.\u003cbr\u003e\n![ddl](resources/ddl_support.png)\n\n# Global ID\n---\nIn Aliyun internal version, global id(also called gid) is supported which marks the id of the database. It can be used to avoid loop when two databases become backup of each other. Mongo-Shake only fetches the oplogs equal to source database id, all the oplogs are be fetched when no gid gave. For current opensource version, it's not supported limited by the modification of MongoDB kernel.\u003cbr\u003e\nIf you want to build active-active replication without `gid` supported, please visit [FAQ document](https://github.com/alibaba/MongoShake/wiki/FAQ) to see more details.\u003cbr\u003e\n\n# Tunnel\n---\nAs mentioned above, we support several tunnel types such like: rcp, tcp, file, kafka, mock and direct. __rpc__ and __tcp__ means connecting to receiver synchronously by net/rcp and TCP respectively; __file__ means writing output into file; __kafka__ is an asynchronous way of sending the message; __mock__ is used by testing that throws away all the data; __direct__ means writing into target MongoDB directly. Users can also add or modify current tunnel type.\u003cbr\u003e\nWe offer receiver to connect to different tunnels like: rpc, tcp, file, mock and kafka. Please visit [FAQ document](https://github.com/aliyun/mongo-shake/wiki/FAQ) to see more details.\n\n# Compressor\n---\nGzip, zlib, deflate compressor are supported in batched oplogs before sending.\n\n# Monitor \u0026 Debug\n---\nUsers could monitor or debug Mongo-Shake through RESTful API, please visit [FAQ document](https://github.com/aliyun/mongo-shake/wiki/FAQ) to see more details.\n\nUsers could also monitor replication metrics with Prometheus exporter. See more details in [mongoshake-prometheus-exporter](https://github.com/osgurisdosre/mongoshake-prometheus-exporter) and [issue#859](https://github.com/alibaba/MongoShake/issues/859)\n\nSince v2.8.8, MongoShake also support prometheus metrics. See more details in [README](/docs/monitor). The following is an example:\n![monitor_prometheus_example2](resources/monitor_prometheus_example1.png )\n![monitor_prometheus_example2](resources/monitor_prometheus_example2.png )\n\n# Other Details\n---\nMongo-Shake uses [go-driver](github.com/mongodb/mongo-go-driver)  to fetch oplogs from source MongoDB which is later than the given timestamp in configuration. Then, it filters oplogs based on whitelist, blacklist, and gid. All the oplogs will be transferred at least once which is acceptable because of idempotent of oplog DML. We use __seq__ and __ack__ to make sure the package is received which is similar to the sequence and acknowledgment numbers in TCP.\u003cbr\u003e\nThe oplogs are batched together in the handling pipeline.\u003cbr\u003e\nUsers can adjust the worker concurrency and executor concurrency according to the different environment.\u003cbr\u003e\nPlease see the detail documents listed at the beginning if you want to see more details.\u003cbr\u003e\n\n# Code branch rules\nversion rules: a.b.c.\n\n*  a: major version\n*  b: minor version. **even number means stable version**. e.g. 1.2.x, 1.4.x, 2.0.x are stable while 1.5.x, 2.1.x aren't.\n*  c: bugfix version\n\n| branch name | rules |\n| - | :- |\n| master | master branch, do not allowed push code. store the latest stable version. |\n| **develop**(main branch) | develop branch. all the bellowing branches fork from this. |\n| feature-\\* | new feature branch. forked from develop branch and then merge back after finish developing, testing, and code review. |\n| bugfix-\\* | bugfix branch. forked from develop branch and then merge back after finish developing, testing, and code review. |\n| improve-\\* | improvement branch. forked from develop branch and then merge back after finish developing, testing, and code review.  |\n\ntag rules:\nadd tag when releasing: \"release-v{version}-{date}\". for example: \"release-v1.0.2-20180628\"\n\n# Usage\n---\nRun `./bin/collector.darwin` or `collector.linux` which is built in OSX and Linux respectively.\n\nOr you can build mongo-shake yourself according to the following steps(go version needs \u003e= 15.10):\n\n*  git clone https://github.com/alibaba/MongoShake.git\n*  cd MongoShake\n*  make\n*  ./bin/collector -conf=conf/collector.conf \n\nplease note: user must modify collector.conf first to match needs. You can also use \\\"start.sh\\\" script which supports hypervisor mechanism in Linux OS only.\n\n# Shake series tool\n---\nWe also provide some tools for synchronization in Shake series.\u003cbr\u003e\n\n* [MongoShake](https://github.com/aliyun/MongoShake): mongodb data synchronization tool.\n* [RedisShake](https://github.com/aliyun/RedisShake): redis data synchronization tool.\n* [RedisFullCheck](https://github.com/aliyun/RedisFullCheck): redis data synchronization verification tool.\n\n# Thanks\n---\n| Username |         Mail          |\n| :------: |:---------------------:|\n| lydarkforest | linyunads1379@163.com |\n| diggzhang |  diggzhang@gmail.com  |\n| ManleyLiu |    daywbdb@qq.com     |\n| hustchensi |   chensi_04@126.com   |\n| HelloCodeMing | huanmingwong@163.com  |\n| cocoakekeyu | cocoakekeyu@gmail.com |\n| lixj1103 |   244769542@qq.com    |\n| xzshinan | shinan@gongchang.com  |\n| tzjavadmg |   codyzeng@163.com    |\n| dx8439 |   171390022@qq.com    |\n| monkeyWie |                       |\n| raydy.yan |  yajuyan@hotmail.com  |\n| loda507 |   741536172@qq.com    |\n| 骑着蜗牛的兔子 |   348978774@qq.com    |\n| lijwww |   2530877879@qq.com   |\n| nanmu42 |      i@nanmu.me       |\n| zemul | zemiaozhou@gmail.com  |\n| renheqiang |                       |\n| dobesv |   dobesv@gmail.com    |\n| pengzhenyi2015 | 503282373@qq.com |\n| SisyphusSQ | |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falibaba%2Fmongoshake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falibaba%2Fmongoshake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falibaba%2Fmongoshake/lists"}