{"id":14956797,"url":"https://github.com/caosiyang/py-mongo-sync","last_synced_at":"2025-07-13T17:34:23.753Z","repository":{"id":71117289,"uuid":"13520162","full_name":"caosiyang/py-mongo-sync","owner":"caosiyang","description":"Oplog-based data sync tool that synchronizes data from a replica set to another deployment, e.g.: standalone, replica set, and sharded cluster.","archived":false,"fork":false,"pushed_at":"2023-09-25T18:54:19.000Z","size":199,"stargazers_count":109,"open_issues_count":2,"forks_count":46,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-05T04:51:13.531Z","etag":null,"topics":["data-migration","data-transmission","mongo","mongodb","oplog","oplog-based","python","realtime","sync"],"latest_commit_sha":null,"homepage":"http://caosiyang.github.io/py-mongo-sync/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/caosiyang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-10-12T10:28:35.000Z","updated_at":"2025-02-03T10:10:20.000Z","dependencies_parsed_at":"2024-09-02T15:41:41.298Z","dependency_job_id":null,"html_url":"https://github.com/caosiyang/py-mongo-sync","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/caosiyang/py-mongo-sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caosiyang%2Fpy-mongo-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caosiyang%2Fpy-mongo-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caosiyang%2Fpy-mongo-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caosiyang%2Fpy-mongo-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/caosiyang","download_url":"https://codeload.github.com/caosiyang/py-mongo-sync/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caosiyang%2Fpy-mongo-sync/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265179206,"owners_count":23723425,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-migration","data-transmission","mongo","mongodb","oplog","oplog-based","python","realtime","sync"],"created_at":"2024-09-24T13:13:33.059Z","updated_at":"2025-07-13T17:34:23.734Z","avatar_url":"https://github.com/caosiyang.png","language":"Python","readme":"# py-mongo-sync\n\nIt synchronizes data from a replica set to another MongoDB deployment, e.g., standalone, replica set, and sharded cluster.\n\nIt's oplog-based and provides a realtime data synchronization.\n\nIt's written in Python 2.7.\n\n## Support\n\n- MongoDB 2.4\n- MongoDB 2.6\n- MongoDB 3.0\n- MongoDB 3.2\n- MongoDB 3.4\n\n## Features\n\n- initial sync and oplog based incremental sync\n- sync the specified databases and collections\n- concurrent oplog replaying\n\n## Requirements\n\nSee [requirements](./requirements.txt) for details.\n\n- gevent\n- toml\n- mmh3\n- pymongo\n\n    Always use pymongo 3.5.1.\n\n    Refer to [https://api.mongodb.com/python/3.6.0/changelog.html](https://api.mongodb.com/python/3.6.0/changelog.html)\n\n    \u003e Version 3.6 adds support for MongoDB 3.6, drops support for CPython 3.3 (PyPy3 is still supported), and drops support for MongoDB versions older than 2.6. If connecting to a MongoDB 2.4 server or older, PyMongo now throws a ConfigurationError.\n\n## Notice\n\n- source **MUST** be a replica set\n- ignore system databases\n    - admin\n    - local\n- ignore system collections\n    - system.\\*\n- create users for destination manually if necessary\n- suggest to authenticate with administrator if source enabled authentication\n- not support geospatial index\n\nif the source is a sharded cluster\n\n- first, stop the balancer\n- then, start a seprate sync process for each shard\n\n## Configurations\n\nUse [TOML](https://github.com/toml-lang/toml) as configuration file format.\n\nRefer to [mongo_conf.toml](example/mongo_conf.toml).\n\n### src\n\nSource config items.\n\n- src.hosts - hostportstr of a member of replica set\n- src.username - username\n- src.password - password\n- src.authdb - authentiction database\n\n### dst\n\nDestination config items.\n\n- dst.mongo.hosts\n- dst.mongo.authdb\n- dst.mongo.username\n- dst.mongo.password\n\n### sync\n\nCustom options for synchronization.\n\n`sync.dbs` specfies the databases to sync.\n`sync.dbs.colls` specifies the collections to sync.\n\n- sync.dbs - databases to sync, sync all databases if not specify\n    - sync.dbs.db - source database name\n    - sync.dbs.rename_db - destination database name, stay the same if not specify\n    - sync.dbs.colls - collectons to sync, sync all collections if not specify\n\n`coll` in `sync.dbs.colls` element specifies the collection to sync.\n`fileds` in `sync.dbs.colls` element specifies the fields of current collection to sync.\n\n### log\n\n- log.filepath - log file path, write to stdout if empty or not set\n\n## Usage \n\nCommand options has functional limitations.\nIt's strongly recommended that use config file.\n\n### sync\n\n```bash\nusage: sync.py [-h] [-f [CONFIG]] [--src [SRC]] [--src-authdb [SRC_AUTHDB]]\n               [--src-username [SRC_USERNAME]] [--src-password [SRC_PASSWORD]]\n               [--dst [DST]] [--dst-authdb [DST_AUTHDB]]\n               [--dst-username [DST_USERNAME]] [--dst-password [DST_PASSWORD]]\n               [--start-optime [START_OPTIME]]\n               [--optime-logfile [OPTIME_LOGFILE]] [--logfile [LOGFILE]]\n\nSync data from a replica-set to another MongoDB/Elasticsearch.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -f [CONFIG], --config [CONFIG]\n                        configuration file, note that command options will\n                        override items in config file\n  --src [SRC]           source should be hostportstr of a replica-set member\n  --src-authdb [SRC_AUTHDB]\n                        src authentication database, default is 'admin'\n  --src-username [SRC_USERNAME]\n                        src username\n  --src-password [SRC_PASSWORD]\n                        src password\n  --dst [DST]           destination should be hostportstr of a mongos or\n                        mongod instance\n  --dst-authdb [DST_AUTHDB]\n                        dst authentication database, default is 'admin', for\n                        MongoDB\n  --dst-username [DST_USERNAME]\n                        dst username, for MongoDB\n  --dst-password [DST_PASSWORD]\n                        dst password, for MongoDB\n  --start-optime [START_OPTIME]\n                        timestamp in second, indicates oplog based increment\n                        sync\n  --optime-logfile [OPTIME_LOGFILE]\n                        optime log file path, use this as start optime if\n                        without '--start-optime'\n  --logfile [LOGFILE]   log file path\n\n```\n\n## TODO List\n\n- [ ] command options tuning\n- [ ] config file format tuning\n- [ ] sync sharding config (enableSharding \u0026 shardCollection)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaosiyang%2Fpy-mongo-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaosiyang%2Fpy-mongo-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaosiyang%2Fpy-mongo-sync/lists"}